{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在bigram中我们仅考虑了预测前一个单词的概率，并且我们不能学习到不同词汇之间的相似度，这一点很重要。\n",
    "例如\"a dog is running in the room\"我们可能会学习到\"dog\"与\"cat\"之间是非常相近的，从而轻松将其替换"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"./names.txt\",\"r\",encoding=\"utf-8\") as f:\n",
    "    names = f.read().splitlines()\n",
    "print(names[:10])\n",
    "print(len(names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chars = sorted(set(\"\".join(names)))\n",
    "ctoi = {ch:i+1 for i, ch in enumerate(chars)}\n",
    "ctoi['.'] = 0\n",
    "itoc = {i:c for c, i in ctoi.items()}\n",
    "print(ctoi['.'])\n",
    "print(itoc[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "论文中首次提出了不使用独热编码，而是建立词向量的方法，为每个词建立一个低纬度的向量来表达蕴含的信息，相近的词最终会在空间上相近"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用三个预测下一个，首先应该生成输入和输出的索引  \n",
    "X 是(n, block_size)的二维，Y是(n, block_size)的一维是X的下一个"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# block_size 是训练时序列的长度，开头的地方需要填充\n",
    "block_size = 3 # 用三个预测下一个，首先应该生成输入和输出\n",
    "def build_datasets(names):\n",
    "    X = [] # 网络输入的idxs\n",
    "    Y = [] # 网络输出的idxs\n",
    "    for name in names:\n",
    "        print(name)\n",
    "        name = name + '.'\n",
    "        context = [0] * block_size\n",
    "        for ch in name:\n",
    "            X.append(context)\n",
    "            # print(context)\n",
    "            idx = ctoi[ch]\n",
    "            Y.append(idx)\n",
    "            # print(\"\".join([itoc[id] for id in context]),f\" ---> {itoc[idx]}\")\n",
    "            context = context[1:] + [idx]\n",
    "    X = torch.tensor(X)\n",
    "    Y = torch.tensor(Y)\n",
    "    print(X.shape, Y.shape)\n",
    "    return X, Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将数据集分为训练集、测试集、验证集\n",
    "names_n = len(names)\n",
    "n_80 = int(0.8 * names_n)\n",
    "n_90 = int(0.9 * names_n)\n",
    "train_dataX, train_dataY = build_datasets(names[:n_80])\n",
    "test_dataX, test_dataY = build_datasets(names[n_80:n_90])\n",
    "val_dataX, val_dataY = build_datasets(names[n_90:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 建立单词嵌入表，论文中17000个单词用了30维，我们这里27个字符用2维尝试\n",
    "chars_n = len(chars) + 1\n",
    "char2vec = 2\n",
    "C = torch.randn((chars_n, char2vec)) # 词嵌入表27个字符，二维\n",
    "print(C.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(C.tolist())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "torch可以直接将矩阵作为索引，将索引的内容添加到最低维度上"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 把索引装化成词向量，现在的X转化之后就是(32, 3, 2)\n",
    "def embedded(X):\n",
    "    return C[X]\n",
    "train_emb = embedded(train_dataX)\n",
    "print(train_emb.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](./img/image.png)\n",
    "\n",
    "文中将三个词向量相连作为输入，中间是MLP的隐藏层，输出是单词个数，我们这里使用6，100，17的网络结构，首先将输入转化为（32，6）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "torch的view不会创建新的张量，这就是为什么C向量可以一次一次的训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reshape(emb):\n",
    "    return emb.view(-1, block_size*char2vec)\n",
    "input_train = reshape(train_emb)\n",
    "print(input_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 第一层的权重和偏移，对于tanh要乘以5/3\n",
    "g = torch.Generator().manual_seed(2757539057)\n",
    "W1_in = char2vec*block_size\n",
    "W1_out = 200\n",
    "W1 = torch.randn((W1_in, W1_out),  generator = g) / (W1_in**0.5) * (5/3)\n",
    "W2 = torch.randn((W1_out, chars_n),generator = g) * 0.01\n",
    "b1 = torch.randn(1) * 0\n",
    "b2 = torch.randn(1) * 0\n",
    "# 对于较深的神经网络每一层输入的数据在经过层层运算后可能会得到一些很难看的分布，这些分布使得网络难以收敛，Google提出了批量归一化，将待激活的参数重分布到高斯分布\n",
    "# 需要注意的是标准化之后不一定就是正态分布，只是控制了数据的范围\n",
    "hbias = torch.ones((1, W1_out)) # 整体批量的标准差初始化\n",
    "hgain = torch.zeros((1, W1_out))# 整体批量的平均值初始化\n",
    "hstd_run = torch.ones((1, W1_out)) # 整体批量的标准差初始化\n",
    "hmean_run = torch.zeros((1, W1_out))# 整体批量的平均值初始化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "paramters = [C,W1,b1,W2,b2,hbias,hgain]\n",
    "for p in paramters:\n",
    "    p.requires_grad = True\n",
    "param_n = (sum(p.nelement() for p in paramters))\n",
    "print(param_n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"../../\")\n",
    "import mymodel.utils\n",
    "import mymodel\n",
    "\n",
    "def forward(inputs:torch.Tensor):\n",
    "    hpreact = inputs @ W1 + b1\n",
    "    hmean = hpreact.mean(dim=0, keepdims=True)\n",
    "    hstd = hpreact.std(dim=0, keepdims=True)\n",
    "    hpreact = hgain * (hpreact - hmean) / hstd + hbias\n",
    "\n",
    "    with torch.no_grad():\n",
    "        # 用这两个参数评估单个批量，在评估函数中\n",
    "        hmean_run = 0.999*hmean_run + 0.001*hmean\n",
    "        hstd_run = 0.999*hstd_run + 0.001*hstd\n",
    "\n",
    "    h = torch.tanh(hpreact)\n",
    "    logits = h @ W2 +b2\n",
    "    return logits\n",
    "\n",
    "def train(X, Y, times, lr , showrate):\n",
    "    myshow = mymodel.utils.show([\"loss\"], f\"steps*{showrate}\", \"loss\")\n",
    "    for i in range(times):\n",
    "        idxs = torch.randint(0, X.shape[0],(32,))\n",
    "        inputs = reshape(embedded(X[idxs]))\n",
    "        # h = torch.tanh(inputs @ W1 +b1)\n",
    "        # logits = h @ W2 +b2\n",
    "        logits = forward(inputs)\n",
    "        loss = nn.functional.cross_entropy(logits, Y[idxs])\n",
    "        for p in paramters:\n",
    "            p.grad = None\n",
    "        loss.backward()\n",
    "        with torch.no_grad():\n",
    "            for p in paramters:\n",
    "                p += -lr * p.grad\n",
    "            if i%showrate==0:\n",
    "                myshow.add([loss])\n",
    "            # print(loss)\n",
    "    eval(X, Y)\n",
    "\n",
    "def eval(X, Y):\n",
    "    with torch.no_grad():\n",
    "        inputs = reshape(embedded(X))\n",
    "        # h = torch.tanh(inputs @ W1 +b1)\n",
    "        # logits = h @ W2 +b2\n",
    "        logits = forward(inputs)\n",
    "        loss = nn.functional.cross_entropy(logits, Y)\n",
    "        print(loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train(train_dataX, train_dataY, 30000, 0.01, 1000)\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval(test_dataX, test_dataY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在前几次的训练中我们发现一个问题，loss开始的时候是非常高的，经过对logits的实验发现，logits方差越大loss基本上越高，因此我们要尽量让logits处在0附近很小的范围内，最后一层的b=0，w*0.01什么的"
   ]
  },
  {
   "attachments": {
    "image-2.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAFMCAIAAAARBb8DAAAAAXNSR0IArs4c6QAAIABJREFUeAHtfQ10VdWZ9oGYhPyQECa3opZQ+0mYjsHUWUID7Sp1hYkBWxxoxRVmFKUrNmLt+C0iAgUza76h0urQ1dUKE6uj7QzBD1h4Bb3AzARrhQIaCf+JgoEYNaA3Jld6mwQC+/PjjLsn95x79nvu/jn73vuelaX77v3u9+d5n/3k3B9uDIIXIoAIIAJphoCRZvViuYgAIoAIEBQ+JAEigAikHQIofGnXciwYEUAEUPiQA4gAIpB2CKDwpV3LsWBEABFA4UMOIAKIQNohgMKXdi3HghEBRCAVhO/SpUtdXV19fX0RvBABRAARiET6+vq6urouXboUT+JTQfi6uroMvBABRAARGI5AV1dXKgtfX1+fYRhdXV2+/KoLh8NNTU3hcNiX6FKDYmlS4ZXkHLsWiUTMm6G+vr5UFr5IJGIYRiQSiVek1PkLFy4Eg8ELFy5IjeKLcyzNF9g5g2LXCCFMTUiFp7rMIjmZ5L4deeaOj56r2DU9++KeFbxrTE1A4XOHmr0Kbwbbl2YWWJpmDQGlg13DOz4QUTiNkGecAPqyHbvmC+ycQeFdwzs+TqjZ2+HNYPvSzAJL06whoHSwa3jHByIKpxHyjBNAX7Zj13yBnTMovGt4x8cJNXs7vBlsX5pZYGmaNQSUDnYN7/hAROE0Qp5xAujLduyaL7BzBoV3De/4OKFmb4c3g+1LMwssTbOGgNLBruEdH4gonEbIM04AfdmOXfMFds6g8K7hHR8n1Ozt8GawfWlmgaVp1hBQOtg1Le743n///b/7u78bO3bsqFGjysrK3nzzTbN7ly9fXrVq1bhx40aNGlVZWfnOO+/Qrvb09CxYsGD06NGFhYWLFi06f/48XXIcMNXdcZeoSeSZKCRV+oF0bcKjL9t/VCaZWCxIaYl59n0XvDSmJsj9lxuffPLJhAkT7r333gMHDnR0dOzatevUqVMmfGvWrCksLAwGg4cPH54zZ87111/f399vLlVXV5eXl+/fv//111+/4YYbampq3BFnFum+nXMV3gzOQOq3p3lpdtWb8OjL6rvgNWKad82Ei6kJcoXv0Ucf/cY3vmHv3OXLl8eNG/fEE0+YS319fdnZ2Rs3biSEnDhxwjAMemO4Y8eOESNGfPDBBzFOBgYG6JdbmN/EEA6HL/hxRaPRYDAYjUb9CC43ZpqXVrpiu/1HLuIivKd510wIw+Gw+xeXyBW+r3zlKw8//PD3vve9QCDw1a9+9emnnzb169133zUMo7W1lcrZN7/5zR/96EeEkGeffXbMmDF0/uLFixkZGVu3bqUz5qChoWH4V28ZTU1NQbwQAUQAEQgGm5qa/BS+7CvX8uXLDx482NjYOGrUqOeff54QsnfvXsMwPvzwQypnd9555/z58wkhq1evLi0tpfOEkEAgsG7dOusMIQTv+ETcHDB8pPC9w+SV24LB4OSV2+g9nR0LumQd2M10m0nhrsFL8/mOLzMzc9q0aVSzHnrooYqKCiHCR31C3sGxGgsf40sqwiFV4LB0xfZgMFi6Yjt9Ic8elC5ZB3Yz3WaQkBBNkPtUt6Sk5Pvf/z5lxrp166699lpCCP9TXeoTUqTVWPgYeSYcUgUOUfgUgCw8BPys+fzmRk1NjfXNjYcffti8ATTf3HjyySdNaCKRSMybGy0tLebSrl27HN/csGLKLNJqLHwMb4bw0LIdpnBpKHyyySPDP5yQTE2Qe8f3xhtvXHXVVatXrz558uSGDRtyc3P/4z/+w0RkzZo1Y8aMeemll44cOXLHHXfEfJzl5ptvPnDgwJ49eyZOnIgfZ5HBIYhPOM8g3vyysT5RpWMUPr/awRMXTkifhY8Qsn379rKysuzs7L/8y7+k7+oSQswPMF999dXZ2dmVlZVvv/02RaSnp6empiY/P7+goOC+++7DDzBTZBQP4DxTnJincFTsrAMUPk8YamIMJ6T/wqcAMmaRUnOAN0NqGjKcp0ZpVr2jYxQ+GYSR7RNOSKYmyH2qKxsI0z+zSKlpwJshNQ0ZzlOjNCp21gEKnwzCyPYJJyRTE1D4eJsFbwZvJOX7U6M0q97RMQqfcjYJCAgnJAqfALjdXcCb4e5Hw9XUKI2KnXWAwqch35gpwQmJwscEk9cA3gzeSMr3J2NpVnVzGaPwKWeTgIBwQqLwCYDb3QW8Ge5+NFxNxtJcxM66hMKnId+YKcEJicLHBJPXAN4M3kjK9ydjaVZ1cxmj8Clnk4CAcEKi8AmA290FvBnufjRcTcbSXMTOuoTCpyHfmCnBCYnCxwST1wDeDN5IyvcnY2lWdXMZo/ApZ5OAgHBCovAJgNvdBbwZ7n40XE3G0lzEzrqEwqch35gpwQmJwscEk9cA3gzeSMr3J2NpVnVzGaPwKWeTgIBwQqLwCYDb3QW8Ge5+NFxNxtJcxM66hMKnId+YKcEJicLHBJPXAN4M3kjK9ydjaVZ1cxmj8Clnk4CAcEKi8AmA290FvBnufjRcTcbSXMTOuoTCpyHfmCnBCYnCxwST1wDeDN5IyvcnY2lWdXMZo/ApZ5OAgHBCovAJgNvdBbwZ7n40XE3G0lzEzrpkFz7rqqexbo1Lxq4BMYSXhsIHhDRxM3gzEo/h0079S/OkUFZjFD6fOMUVFk5IFD4uoCGb4c2AeNPKRv/SrFrmaYzCpxXTgMnACYnCB4Q0cTN4MxKP4dNO/UvzJHZWYxQ+nzjFFRZOSBQ+LqAhm+HNgHjTykb/0qxa5mmMwqcV04DJwAmJwgeENHEzeDMSj+HTTq1K86RrTGMUPp84xRUWTkgUPi6gIZvhzYB408pGq9KYWubJAIVPK6YBk4ETEoUPCGniZvBmJB7Dp51aleZJ15jGKHw+cYorLJyQKHxcQEM2w5sB8aaVjValMbXMkwEKn1ZMAyYDJyQKHxDSxM3gzUg8hk87tSrNk64xjVH4fOIUV1g4IVH4uICGbIY3A+JNKxutSmNqmScDFD6tmAZMBk5IFD4gpImbwZuReAyfdmpVmiddYxqj8PnEKa6wcEKi8HEBDdkMbwbEm1Y2WpXG1DJPBih8WjENmAyckCh8QEgTN4M3I/EYPu3UqjRPusY0RuHziVNcYeGEROHjAhqyGd4MiDetbLQqjallngxQ+LRiGjAZOCFR+ICQJm4Gb0biMXzaqVVpnnSNaYzC5xOnuMLCCYnCxwU0ZDO8GRBvWtloVRpTyzwZoPBpxTRgMnBC+ix8DQ0NhuWaNGmSWWF/f//ixYvHjh2bl5c3b968s2fP0so7Oztnz56dk5MTCATq6+svXrxIl+INmEXG2yhkHt4MIeFUOtGqNE+6xjRG4VNJJFGx4IRkaoIhKidHPw0NDTfeeGP359fHH39smtXV1Y0fP765ubmlpaWiomL69Onm/NDQUFlZ2cyZM1tbW0OhUHFx8fLlyx09WyeZRVqNhY/hzRAeWrZDrUpjapknAxQ+2eSR4R9OSKYmSBe+8vLyGAj6+voyMzM3b95szre1tRmGsW/fPkJIKBQaOXIkvQFcv359QUHB4OBgjIeYh8wiY+zFPoQ3Q2xcBd60Ks2TrjGNUfgU8Ed4CDghmZogXfhyc3Ovueaa66+/fsGCBZ2dnYSQ5uZmwzB6e3spLiUlJWvXriWErFq1yiqUHR0dhmEcPHiQWtLBwMBA5POrq6vLMIxwOHzBjysajQaDwWg06kdwuTG1Kq10xXaBP5NXbgsGg5NXbuP3KbcH3r1r1TXv6bvtgJcWDocNw4hEIlQxYgZyhS8UCm3atOnw4cM7d+6cNm1aSUnJp59+umHDhqysLGseU6ZMWbp0KSGktra2qqqKLkWjUcMwQqEQnaGDmFcPDcNoamoK4oUIIAKIQDDY1NTkp/BRnSKE9Pb2FhQUPPPMM0KED+/43H4zClqD/4IVFNDNDf+tmdUD3vG5Ya3rGpyQPt/xWYWPEHLLLbcsW7ZMyFNdq2fm83mrsfAx/HUH4aFlO9SqNObLdp4M8DU+2eSR4R9OSKYmyH2qay3+/PnzRUVFv/jFL8w3N7Zs2WKutre3x7y5ce7cOXOpsbGxoKBgYGDA6sc+ZhZp3yJwBt4MgUHVuNKqNE+6xjRG4VNDIbFR4IRkaoJc4VuyZMnvfve706dP7927d+bMmcXFxR999BEhpK6urqSkZPfu3S0tLdOuXCZA5sdZqqqqDh06tHPnzkAggB9nEUsdT97gPPPkNjFjppZ5MkDhS6wL/u6CE9Jn4bvrrruuueaarKys66677q677jp16pQJnPkB5qKiotzc3Llz53Z3d1NAz5w5M2vWrJycnOLi4iVLluAHmCky6gdwninIzZOuMY1R+BS0THgIOCF9Fj7hlTs6ZBbpuEvUJLwZoiIq86NVaUwt82SAwqeMRQIDwQnJ1AS5T3UF1uziilmky17+JXgz+GMp9qBVaZ50jWmMwqeYS0LCwQnJ1AQUPt6OwJvBG0n5fq1KY2qZJwMUPuVsEhAQTkgUPgFwu7uAN8Pdj4arWpXmSdeYxih8GvKNmRKckCh8TDB5DeDN4I2kfL9WpTG1zJMBCp9yNgkICCckCp8AuN1dwJvh7kfDVa1K86RrTGMUPg35xkwJTkgUPiaYvAbwZvBGUr5fq9KYWubJAIVPOZsEBIQTEoVPANzuLuDNcPej4apWpXnSNaYxCp+GfGOmBCckCh8TTF4DeDN4Iynfr1VpTC3zZIDCp5xNAgLCCYnCJwBudxfwZrj70XBVq9I86RrTGIVPQ74xU4ITEoWPCSavAbwZvJGU79eqNKaWeTJA4VPOJgEB4YRE4RMAt7sLeDPc/Wi4qlVpnnSNaYzCpyHfmCnBCYnCxwST1wDeDN5IyvdrVRpTyzwZoPApZ5OAgHBCovAJgNvdBbwZ7n40XNWqNE+6xjQWKHz2WP62UquuiYUCXhoKn1jkHbzBm+GwWe8prUqz6wvPDAqf3tRzzg5OSBQ+ZwQFzsKbITCoGldalcYjc/a9KHxqKCQ2CpyQKHxikXfwBm+Gw2a9p7QqzS5ePDMofHpTzzk7OCFR+JwRFDgLb4bAoGpc+Vgaj6hB9qLwqaGQ2ChwQqLwiUXewRu8GQ6b9Z7ysTSIePHYoPDpTT3n7OCEROFzRlDgLLwZAoOqceVjaTyiBtmLwqeGQmKjwAmJwicWeQdv8GY4bNZ7ysfSIOLFY4PCpzf1nLODExKFzxlBgbPwZggMqsaVj6XxiBpkLwqfGgqJjQInJAqfWOQdvMGb4bBZ7ykfS4OIF48NCp/e1HPODk5IFD5nBAXOwpshMKgaVz6WxiNqkL0ofGooJDYKnJAofGKRd/AGb4bDZr2nfCwNIl48Nih8elPPOTs4IVH4nBEUOAtvhsCgalz5WBqPqEH2ShU+xwTUtIwQ4mPXZNcILw2FT3YvkGdSEHbUDoGTKHxS2ibZKQrfMICZ6j7MWvQDeDNER5buT1lpAhUN6AqFTzp7JASAE5KpCYaE9FS7ZBYpNSF4M6SmIcO5stKAaiXQDIVPBmFk+4QTkqkJKHy8zYI3gzeS8v3KShOoaEBXKHzK2SQgIJyQKHwC4HZ3AW+Gux8NV5WVBlQrgWYofBryjZkSnJAofEwweQ3gzeCNpHy/stIEKhrQFQqfcjYJCAgnpC7C9/jjjxuG8Q//8A9m9f39/YsXLx47dmxeXt68efPOnj1LUens7Jw9e3ZOTk4gEKivr7948SJdijdgFhlvo5B5eDOEhFPpRFlpQLUSaIbCp5JIomLBCcnUBBWv8b3xxhtf+tKXbrrpJip8dXV148ePb25ubmlpqaiomD59ugnN0NBQWVnZzJkzW1tbQ6FQcXHx8uXLmagxi2R64DGAN4Mnii97lZUmUNGArlD4fGEUZ1A4IZmaIF34zp8/P3HixP/6r/+aMWOGKXx9fX2ZmZmbN282UWhrazMMY9++fYSQUCg0cuRIegO4fv36goKCwcFBd7yYRbpv51yFN4MzkPrtykoDqpVAMxQ+9XTijwgnJFMTpAvfPffc8/DDDxNCqPA1Nzd/9rS3t7eXAlFSUrJ27VpCyKpVq8rLy+l8R0eHYRgHDx6kM3QwMDAQ+fzq6uoyDCMcDl/w44pGo8FgMBqN+hFcbkxlpZWu2K74Z/LKbcFgcPLKbcriym2VxbuyrlliKhrCSwuHw4ZhRCIRqhgxA7nCt3HjxrKysv7+fqvwbdiwISsry5rHlClTli5dSgipra2tqqqiS9Fo1DCMUChEZ+igoaHBGH41NTUF8UIEEAFEIBhsamryTfjee++9L3zhC4cPHzbVit7xCRE+vONT8EsW/guWMxllt100EN7xcbbMl+1wQvp5x/fiiy8ahpHx+WUYxogRIzIyMv77v/+b/6kuvfUjhDCfz1uNhY/hrzsIDy3bobLSBL54B3SFr/HJJo8M/3BCMjVB4lPdTz/99KjluuWWW/7+7//+6NGj5psbW7ZsMaFpb2+PeXPj3Llz5lJjY2NBQcHAwIA7iMwi3bdzrsKbwRlI/XZlpQHVSqAZCp96OvFHhBOSqQkShS+mTvpUlxBSV1dXUlKye/fulpaWaVcu09j8OEtVVdWhQ4d27twZCATw4ywxMKp8COcZZ1YCFQ3oCoWPs2W+bIcTUlPhMz/AXFRUlJubO3fu3O7uborjmTNnZs2alZOTU1xcvGTJEvwAM0VG/QDOM87cgGol0AyFj7NlvmyHE1Ij4ZOHFLNIeaHxex+FYCtQ0YCuUPiENE6xExS+YYCj8A2DQ9wDOM84YwLVSqCZeuGzJ88JWrztyroWLwF58/DSmJqg7jU+eXAwi5QXGu/4hGBrFwXZMyh8Qhqn2AkK3zDAUfiGwSHuAZxnnDFly5zdPwofZ8t82Q4nJFMT8I6Pt4PwZvBGUr5fWWl2YZI9g8KnnE0CAsIJicInAG53F/BmuPvRcFVZabJlzu4fhU9DvjFTghMShY8JJq8BvBm8kZTvV1aaXZhkz6DwKWeTgIBwQqLwCYDb3QW8Ge5+NFyVUZpsRQP6R+HTkG/MlOCEROFjgslrAG8GbyTl+2WUBhQm2WYofMrZJCAgnJAofALgdncBb4a7Hw1XZZQmW9GA/lH4NOQbMyU4IVH4mGDyGsCbwRtJ+X4ZpQGFSbYZCp9yNgkICCckCp8AuN1dwJvh7kfDVRmlyVY0oH8UPg35xkwJTkgUPiaYvAbwZvBGUr5fRmlAYZJthsKnnE0CAsIJicInAG53F/BmuPvRcFVGabIVDegfhU9DvjFTghMShY8JJq8BvBm8kZTvl1EaUJhkm6HwKWeTgIBwQqLwCYDb3QW8Ge5+NFyVUZpsRQP6R+HTkG/MlOCEROFjgslrAG8GbyTl+2WUBhQm2WYofMrZJCAgnJAofALgdncBb4a7Hw1XZZQmW9GA/lH4NOQbMyU4IVH4mGDyGsCbwRtJ+X4ZpQGFSbYZCp9yNgkICCekMOF7/vnnX375ZTP3Rx55pLCwcNq0aWfOnBFQDbcLZpHcEdwcwJvh5kXLNRmlyVY0oH8UPi0Zx0gKTkimJkC/j6+0tLS5uZkQ8oc//CE3N7exsfE73/nO3LlzGZkqWWYWKTULeDOkpiHDuYzSgMIk2wyFTwZhZPuEE5KpCVDhy8nJ6ezsJIQsXbr07rvvJoQcO3asuLhYdqkQ/8wiIU4StoE3I+EQfm2UUZpsRQP6R+Hzi1Q8ceGEZGoCVPgCgcDBgwcJIV/96ld/+9vfEkJOnTqVl5fHU4aovcwiRQVy9ANvhuN2nSdllAYUJtlmKHw6Ey9ebnBCMjUBKnwLFiz467/+6+9///u5ubnhcJgQ8tJLL914443xUlQ5zyxSajLwZkhNQ4ZzGaXJVjSgfxQ+GYSR7RNOSKYmQIWvt7f3wQcfnDNnzo4dO8zyHnvssX/+53+WXSrEP7NIiJOEbeDNSDiEXxtllAYUJtlmKHx+kYonLpyQTE2ACh9PurL3MouUmgC8GVLTkOFcRmmyFQ3oH4VPBmFk+4QTkqkJUOHbsWPH66+/bhb2q1/9qry8vKam5pNPPpFdKsQ/s0iIk4Rt4M1IOIRfG2WUBhQm2WYofH6RiicunJBMTYAKX1lZ2SuvvEIIOXLkSHZ29vLlyysqKu69916eMkTtZRYpKpCjH3gzHLfrPCmjNNmKBvSPwqcz8eLlBickUxOgwpeXl3f69OnPEmpoaPjud79LCHnrrbeuvvrqeCmqnGcWKTUZeDOkpiHDuYzSgMIk2wyFTwZhZPuEE5KpCVDhKyoqOn78OCHk61//emNjIyHk9OnTOTk5skuF+GcWCXGSsA28GQmH8GujjNJkKxrQPwqfX6TiiQsnJFMToML3ne9857bbbvunf/qnzMzM999/nxCya9euiRMn8pQhai+zSFGBHP3Am+G4XedJGaUBhUm2mQ7CZ69RCBlkdE1IYvxO4KUxNQEqfJ2dnbfffvtNN930zDPPmAU8/PDDDz30EH8x/B6YRfKHcPEAb4aLEz2XZJRmP+2+zKDw6Uk596zghGRqAlT43BOKt7pu3brJkyePvnJVVFSEQiHTsr+/f/HixWPHjs3Ly5s3b97Zs2eph87OztmzZ+fk5AQCgfr6+osXL9KleANmkfE2CpmHN0NIOJVOZJTmi8zZg6LwqSSSqFhwQjI1wYPwDQ0Nbdmy5f9cubZu3To0NMSsZ9u2ba+88so777zz9ttvr1ixIjMz89ixY4SQurq68ePHNzc3t7S0VFRUTJ8+3XT1WYiysrKZM2e2traGQqHi4uLly5czozCLZHrgMYA3gyeKL3v5S7MrjiYzKHy+MIozKJyQTE2ACt/JkycnTpyYm5t785UrNzd30qRJp06d8lRJUVHRM88809fXl5mZuXnzZnNvW1ubYRj79u0jhIRCoZEjR9IbwPXr1xcUFAwODrpHYRbpvp1zFd4MzkDqt/OXponM2dNA4VNPJ/6IcEIyNQEqfLNmzaquru7p6TGzD4fD1dXVs2fPBhYzNDS0cePGrKys48ePNzc3G4bR29tL95aUlKxdu5YQsmrVqvLycjrf0dFhGIb55Qh00hwMDAxEPr+6uroMwwiHwxf8uKLRaDAYjEajfgSXG5O/tNIV2/X8mbxyWzAYnLxym1bpCWknf9eEpCHDCby0cDhsGEYkEonRDfoQKny5ublHjhyh2wghhw4dgnw7y5EjR/Ly8jIyMgoLC82PQG/YsCErK8vqasqUKUuXLiWE1NbWVlVV0aVoNGoYBn1lkM5/NmhoaDCGX01NTUG8EAFEABEIBpuamsQIX1FR0d69e63Ss2fPnqKiIuuM43hwcPDkyZMtLS3Lli0rLi4+fvy4EOHDOz4Zv1FjfMJ/wcZspA+1up+yJoN3fLRHSTSAE1LYHd/dd99944037t+///KVa9++fWVlZQsXLnQUu3iTlZWV999/v5CnutYQzOfzVmPhY/jrDsJDy3bIX5r9xTVNZvA1PtnkkeEfTkimJkCf6vb29s6ZM2fEiBFZV64RI0b87d/+rfV1Okidt95668KFC803N7Zs2WJuaW9vj3lz49y5c+ZSY2NjQUHBwMCAu3Nmke7bOVfhzeAMpH47f2mayJw9DRQ+9XTijwgnJFMToMJnJn3y5MltV66TJ09Cyli2bNlrr712+vTpI0eOLFu2bMSIEf/5n/9pfpylpKRk9+7dLS0t065cpjfz4yxVVVWHDh3auXNnIBDAj7NAcJZkA+dZvATsiqPJDApfvJbpPA8nJK/w/W/W5Q7TokWLJkyYkJWVFQgEKisrTdUjhJgfYC4qKsrNzZ07d253dzf1c+bMmVmzZuXk5BQXFy9ZsgQ/wEyRUT+A8yxebprInD0NFL54LdN5Hk5IXuH7lut166236gATs0ipScKbITUNGc75S7MrjiYzKHwyCCPbJ5yQTE3w9lRXdmGJ+WcWmZhb4C54M4AO9THjL00TmbOngcKnD83gmcAJydQEFD447M6W8GY479d4lr80u+JoMoPCpzHv4qYGJyQKX1wQRS3AmyEqojI//KVpInP2NFD4lLFIYCA4IVH4BMLu7AreDOf9Gs/yl2ZXHE1mUPg05l3c1OCEROGLC6KoBXgzREVU5oe/NE1kzp4GCp8yFgkMBCckCp9A2J1dwZvhvF/jWf7S7IqjyQwKn8a8i5sanJAofHFBFLUAb4aoiMr88JemiczZ00DhU8YigYHghEThEwi7syt4M5z3azzLX5pdcTSZQeHTmHdxU4MTEoUvLoiiFuDNEBVRmR/+0jSROXsaKHzKWCQwEJyQKHwCYXd2BW+G836NZ/lLsyuOJjMofBrzLm5qcEKi8MUFUdQCvBmiIirzw1+aJjJnTwOFTxmLBAaCExKFTyDszq7gzXDer/Esf2l2xdFkBoVPY97FTQ1OSBS+uCCKWoA3Q1REZX74S9NE5uxpoPApY5HAQHBCovAJhN3ZFbwZzvs1nuUvza44msyg8GnMu7ipwQmJwhcXRFEL8GaIiqjMD39pmsicPQ0UPmUsEhgITkgUPoGwO7uCN8N5v8az/KXZFUeTGRQ+jXkXNzU4IVH44oIoagHeDFERlfnhL00TmbOngcKnjEUCA8EJicInEHZnV/BmOO/XeJa/NLviaDKDwqcx7+KmBickCl9cEEUtwJshKqIyP/ylaSJz9jRQ+JSxSGAgOCFR+ATC7uwK3gzn/RrP8pdmVxxNZlD4NOZd3NTghEThiwuiqAV4M0RFVOaHvzRNZM6eBgqfMhYJDAQnJAqfQNidXcGb4bxf41n+0uyKo8kMCp/GvIubGpyQKHxxQRS1AG+GqIjK/PCXponM2dNA4VPGIoGB4IRE4RMIu7MreDOc92s8y1+aXXE0mUHh05h3cVODExKFLy6IohbgzRAVUZkf/tI0kTl7GnoKnz3PCY++7LXd/F3zGlGZPbw0FD7pTYE3Q3oqogPwl+Z4knWYROETTRYV/uCEROGT3g94M6SnIjoAf2k6aJxh6u3EAAAbZElEQVRjDih8osmiwh+ckCh80vsBb4b0VEQH8Fqao8ToOYnCJ5osKvzBCYnCJ70f8GZIT0V0AK+l6alxjlmh8Ikmiwp/cEKi8EnvB7wZ0lMRHcBraY4So+ckCp9osqjwByckCp/0fsCbIT0V0QG8lqanxjlmhcInmiwq/MEJicInvR/wZkhPRXQAr6U5Soyekyh8osmiwh+ckD4L309+8pNbbrklPz8/EAjccccd7e3tFJ7+/v7FixePHTs2Ly9v3rx5Z8+epUudnZ2zZ8/OyckJBAL19fUXL16kS44DZpGOu0RNwpshKqIyP15L01PjHLNC4VPGIoGB4IRkaoIhMC27q9tuu+255547duzYoUOHZs+eXVJS8sc//tE0q6urGz9+fHNzc0tLS0VFxfTp0835oaGhsrKymTNntra2hkKh4uLi5cuX2z1bZ5hFWo2Fj+HNEB5atkOvpTlKjJ6TKHyyySPDP5yQTE2QK3zW4j/66CPDMF577TVCSF9fX2Zm5ubNm02DtrY2wzD27dtHCAmFQiNHjqQ3gOvXry8oKBgcHLS6IoQMDAxEPr+6uroMwwiHwxf8uKLRaDAYjEajfgSXG9NraaUrtifLz+SV24LB4OSV2/RP2GuPvXbNq38f7eGlhcNhwzAikUiMbtCH6oTv5MmThmEcPXqUENLc3GwYRm9vL82jpKRk7dq1hJBVq1aVl5fT+Y6ODsMwDh48SGfMQUNDgzH8ampqCuKFCCACiEAw2NTUpIXwXbp06fbbb//6179uytaGDRuysrKsWjZlypSlS5cSQmpra6uqquhSNBo1DCMUCtEZc4B3fAp+8cJ/wZrJ6H/3RDPEOz4F/BEeAk5IXe746urqJkyY0NXVZcoWv/BZdZD5fN5qLHwMf91BeGjZDr2WpufLeY5Z4Wt8sskjwz+ckExNUPFU98EHH/ziF7/Y0dFBseB/qktdEUKYRVqNhY/hzRAeWrZDr6U5Soyekyh8sskjwz+ckExNkCt8ly9ffvDBB6+99tp33nnHCoT55saWLVvMyfb29pg3N86dO2cuNTY2FhQUDAwMWLfHjJlFxtiLfQhvhti4Crx5LU1PjXPMCoVPAX+Eh4ATkqkJcoXvgQceKCws/N3vftf9+fWnP/3JhKOurq6kpGT37t0tLS3TrlzmvPlxlqqqqkOHDu3cuTMQCODHWYQTCOgQzjPToaPE6DmJwgfkgFZmcEL6LHzD33f9/4+ee+45E0rzA8xFRUW5ublz587t7u6mEJ85c2bWrFk5OTnFxcVLlizBDzBTZBQP4DwzE9NT4xyzQuFTzCUh4eCE9Fn4hFTLdMIskumBxwDeDJ4ovuz1WpqjxOg5icLnC6M4g8IJydQEuU91OesEbmcWCfSTmBm8GYn593GX19L01DjHrFD4fORVwqHhhGRqAgpfwl34n43wZvBGUr7fpTRHNUmiSRQ+5WwSENCFkDHeUfhiABH/EN4M8bEle3QpLYk0zjHVpBY+e0VWIrh0zWqWjGN4aSh80vsLb4b0VEQHcCnNfvaSawaFTzRZVPhzIWRMeBS+GEDEP4Q3Q3xsyR5dSksumbNni8InmTtS3LsQMiYeCl8MIOIfwpshPrZkjy6l2aUkuWZQ+CRzR4p7F0LGxEPhiwFE/EN4M8THluzRpbTkkjl7tih8krkjxb0LIWPiofDFACL+IbwZ4mNL9uhSml1KkmsGhU8yd6S4dyFkTDwUvhhAxD+EN0N8bMkeXUpLLpmzZ4vCJ5k7Uty7EDImHgpfDCDiH8KbIT62ZI8updmlJLlmkkj4IMBaieDSNatZMo7hpaHwSe8vvBnSUxEdwKU0yGnU2QaFTzRZVPhzIWRMeBS+GEDEP4Q3Q3xsyR5paTpLWGK5ofBJ5o4U95SQTO8ofEyIeA3gzeCNpHw/LS0xcdF5FwqfcjYJCEgJyfSFwseEiNcA3gzeSMr309J0lrDEckPhU84mAQEpIZm+UPiYEPEawJvBG0n5flpaYuKi8y4UPuVsEhCQEpLpC4WPCRGvAbwZvJGU76el6SxhieWGwqecTQICUkIyfaHwMSHiNYA3gzeS8v20tMTEReddKHzK2SQgICUk0xcKHxMiXgN4M3gjKd9PS9NZwhLLDYVPOZsEBKSEZPpC4WNCxGsAbwZvJOX7aWmJiYvOu1JM+KxQ09KU80V6QEpIZiQUPiZEvAbwZvBGUr6flmY9V6kxpuqQGuVYq6ClKeeL9ICUkMxIKHxMiHgN4M3gjaR8Py3Neq5SY0zVITXKsVZBS1POF+kBKSGZkVD4mBDxGsCbwRtJ+X5amvVcpcaYqkNqlGOtgpamnC/SA1JCMiOh8DEh4jWAN4M3kvL9tDTruUqNMVWH1CjHWgUtTTlfpAekhGRGQuFjQsRrAG8GbyTl+2lp1nOVGmOqDqlRjrUKWppyvkgPSAnJjITCx4SI1wDeDN5IqvbTU0SPEJ1JmUE6lKaKL+riwM8aCp/0rsCbIT0VQQGouqWDOtBiU2ZAuyaIDhq5gZ81FD7pbYM3Q3oqggJQCaBHiM6kzCAdShNEB43cwM8aCp/0tsGbIT0VQQGouqWDOtBiU2ZAuyaIDhq5gZ81FD7pbYM3Q3oqggJQCaBHiM6kzCAdShNEB43cwM8aCp/0tsGbIT0VQQGouqWDOtBiU2ZAuyaIDhq5gZ81FD7pbYM3Q3oqggJQCaBHiM6kzCAdShNEB43cwM+az8L32muvffvb377mmmsMw3jxxRcphJcvX161atW4ceNGjRpVWVn5zjvv0KWenp4FCxaMHj26sLBw0aJF58+fp0vxBswi420UMg9vhpBwCpxQdUsHdaDFpsyAdk0BVRSHgJ81piYYUlMPhUI//vGPt27dGiN8a9asKSwsDAaDhw8fnjNnzvXXX9/f329mUl1dXV5evn///tdff/2GG26oqalhZsgskumBxwDeDJ4oKvdSCaBHiM6kzCAdSlPJGTWx4GeNqQlyhY/CYRW+y5cvjxs37oknnjBX+/r6srOzN27cSAg5ceKEYRhvvvmmubRjx44RI0Z88MEH1I/jgFmk4y5Rk/BmiIoo2w9Vt3RQB1psygxo12TzRL1/+FljaoIPwvfuu+8ahtHa2kqB++Y3v/mjH/2IEPLss8+OGTOGzl+8eDEjI2Pr1q10hg4GBgYin19dXV2GYYTD4Qt+XNFoNBgMRqNRP4JLiVm6Yrv5M3nltmAwOHnlNjqTMoN0KE0KOXx1Cj9r4XDYMIxIJEIVI2bgg/Dt3bvXMIwPP/yQpnLnnXfOnz+fELJ69erS0lI6TwgJBALr1q2zzpjjhoYGY/jV1NQUxAsRQAQQgWCwqakpNYUP7/jk/eqlt3XpcFtEi02ZAe2aPIb45Tm57/iEPNW13gMyn89bjYWP4a87CA8tySF9tYu+WkRnUmaQDqVJooePbuFnjakJPjzVNd/cePLJJ00EI5FIzJsbLS0t5tKuXbvwzQ31PKPqlg7qQItNmQHtmnrmyI6YNMJ3/vz51iuXYRhr165tbW3t7OwkhKxZs2bMmDEvvfTSkSNH7rjjjpiPs9x8880HDhzYs2fPxIkT8eMssslk908lgB4hOpMyg3Qozd7ZZJ9JGuF79dVXh78DYSxcuJAQYn6A+eqrr87Ozq6srHz77bdpS3p6empqavLz8wsKCu677z78ADNFRtmAqls6qAMtNmUGLl1TRiFJgZJG+CTVH+OW+Xw+xl7sQ3gzxMaV541KgMsRojZJOkjP0uRxRo1n+FljaoKi1/ik4sIsUmp0eDOkpiHQOdWy9FQHWn6SDly6JpAkvriCnzWmJqDw8XYQ3gzeSKr20wPvcoSoTZIO0rM0VQySFQd+1lD4ZPWA+oU3g27RfEC1LD3VgZafpAOXrmlOPGZ68LOGwscEk9cA3gzeSBL2u59tlyPkvlH/1fQsTQKDlLqEnzUUPumNgTdDeireA7grVHqqgzsm+q+6dM07QfTaAT9rKHzSOwdvhvRUvAdwP8YuR8h9o/6r6Vmad4LotQN+1lD4pHcO3gzpqXgP4K5Q6akO7pjov+rSNe8E0WsH/Kyh8EnvHLwZ0lPxHsD9GLscIfeN+q+mZ2neCaLXDvhZQ+GT3jl4M6Sn4j2Au0Klpzq4Y6L/qqeueaeMnzvgZw2FT3qf4M2Qnor3AO7H2NMRcnel2yqW5tIR7zxStAN+1lD4pLcE3gzpqXgP4ML+CY++jOrgjo+eq/xd884jRTvgZw2FT3pL4M2Qnor3AO5Hl/8Iufv3cRVLcwHfO48U7YCfNRQ+6S2BN0N6Kt4DuLAf7/jcwdF2lV/TvfNI0Q74WUPhk94SeDOkpzI8AP/J5D9C/DlI8oCluQA7nEcaPYKfNRQ+6W2DN0N6KsMDuDAbuITqAARKKzP+rg3nkUaP4GcNhU962+DNkJ7K8AD8p5H/CPHnIMkDluYC7HAeafQIftZQ+KS3Dd4M6akMD+DCbOASqgMQKK3M+Ls2nEcaPYKfNRQ+6W2DN0N6KsMD8J9G/iPEn4MkD1iaC7DDeaTRI/hZQ+GT3jZ4M6SnMjyAC7OBS6gOQKC0MuPv2nAeafQIftZQ+KS3Dd4M6akMD8B/GvmPEH8OkjxgaS7ADueRRo/gZw2FT3rb4M2QnsrwAC7MBi6hOgCB0sqMv2vDeaTRI/hZQ+GT3jZ4M6SnMjwA/2nkP0L8OUjygKW5ADucRxo9gp81FD7pbYM3Q3oqwwO4MBu4hOoABEorMxldG84s3x7BzxoKn/QmwZshPZXhAfhPo4wjxJ+VEA9YmicYhzPLt0fws4bCJ71J8GbIS8UTieHGqA5wrPSxlNE1edT15Bl+1lD4PAGbiDG8GYl4h+2RdOpkHCFJqXp1i6V5QgxGQ+lW8LOGwqdRM+Sl4onEcGNUBzhW+ljK6Jojde0lO5oJnEThGwYmU92HWYt+AG+G6Mh/9menoJAZGUdISGL8TrA0Txj+mWqWkd2DZVHKEH7WmJpgSElQrVNmkVLTgTdDXhp2CgqZQXUQAqNiJzK65khde12OZgIn4WeNqQkofLx9gTeDGcnOpAmPvszcRQhx3Mg/KeMI8WclxAOWJgTGGCcQrvLYwM8aCh8PzqC98GYw3cXQyHxo3+VoJmMS1UEGqrJ9+tg1O1fFzsDPWrIK369+9asJEyZkZ2dPnTr1wIED7vAxi3TfzrkKbwYzkOwj4dW/j0fIa6pe7bE0r4hB7JkM5zSAnzWmJuj4VPeFF17Iysr6t3/7t+PHj9fW1o4ZM+bcuXMukDGLdNnLvwRshp039tB2G39nUB38xT+x6D52zU5psTPAs0YIYWqCjsI3derUBx980ITs0qVL11577eOPP+6CILNIl70uS460s9sDm2H3Zndlt/F3xscjJLtwLE02wp7828+C4wzwrCWl8A0ODmZkZLz44ou08nvuuWfOnDn0oTkYGBiIfH699957hmGcPn06DL7+15L/G/Nj3xpjEO/hXz3yQlNT01898kI8g+Sdx9KSsXcp3LXu7u6mpqbu7m77aY2ZOX36tGEYfX19MbpBH2p3x/fBBx8YhvGHP/yBpvjII49MnTqVPjQHDQ0NBl6IACKACMRHoKurK0Y36MNkFT7rHV9vb++7777b19f3+S2g0v93dXUZhtHV1aU0qpJgWJoSmAUHwa5FIpG+vr6urq5Lly5RpYsZaCd8wKe6MWX4+FDSK4w+VkRDY2kUiiQaYNcgzdJO+AghU6dO/eEPf2hmf+nSpeuuu879zQ1InfJskGfysJXnGbsmD1t5ngV2TUfhe+GFF7Kzs59//vkTJ07cf//9Y8aMOXv2rDw0OT0LbAZnJsK3Y2nCIVXgELsGAVlH4SOE/PKXvywpKcnKypo6der+/fshlfhlMzAw0NDQ8Nl//UpAXlwsTR628jxj1yDYaip8kNTRBhFABBCBxBBA4UsMN9yFCCACSYwACl8SNw9TRwQQgcQQQOFLDDfchQggAkmMAApfEjcPU0cEEIHEEEDhSww30tPTs2DBgtGjRxcWFi5atOj8+fN2Rz09PT/84Q9LS0tHjRo1fvz4hx56yOUfD9q3+zUDKY0Q0tjYOGPGjNGjRxuG0dvb61e2zLiQrzjbtGnTpEmTsrOzy8rKXnnlFaZPTQyYpR07dmzevHkTJkwwDOPnP/+5JmlD0mCW9vTTT3/jG98Yc+WqrKxkfnldTFAUvhhAoA+rq6vLy8v379//+uuv33DDDTU1NfadR48enTdv3rZt206dOtXc3Dxx4sTvfve7djPdZiClEUJ+/vOfP37l0ln4IF9xtnfv3oyMjJ/97GcnTpxYuXJlZmbm0aNHdWuKPR9IaW+88UZ9ff3GjRvHjRuXRMIHKW3BggVPPfVUa2trW1vbvffeW1hY+P7779tRijeDwhcPGbf5EydOGIbx5ptvmkY7duwYMWLEBx984LaHkE2bNmVlZV28eNHdzN9Vr6W9+uqrOgsf5CvO5s+ff/vtt1PYv/a1r/3gBz+gD7UdQEqjyU+YMCGJhM9TaYSQoaGh0aNH/+Y3v6H1MgcofEyIHAyeffbZMWPG0IWLFy9mZGRs3bqVzjgOfv3rXxcXFzsu6TPptTSdhQ/4777Hjx9vFYXHHnvspptu0qcjjpkAS6N7k0j4vJZGCPn0009HjRq1fft2Wi9zgMLHhMjBYPXq1aWlpdaFQCCwbt0660zM+OOPPy4pKVmxYkXMvG4PvZams/ABv+IsMzOzqamJNuKpp576whe+QB/qOQCWRpNPIuHzWhoh5IEHHvjyl7/c399P62UOUPicIXr00Ufjfc1XW1ubV3WIRCKf/du76urqCxcuOMdTOCu2NBQ+ha37cyiv6pDCwvf4448XFRUdPnz4z+gARih8ziB99NFHbXGuwcFBT88HP/3002nTplVWVnr6jeSclohZgaURQnQWPuCTJnyqK4JWwnwAu2bGe+KJJwoLC+mr7fAkUPjgWP3Z0nwHoKWlxZzatWtXvDc3IpFIRUXFjBkzotHon/drPIKXZhahs/ABv+Js/vz53/72t2lPpk2blixvbsC/vS2J7viAXSOE/PSnPy0oKNi3bx/tHXyAwgfHaphldXX1zTfffODAgT179kycOJF+nOX999+fNGmS+amiSCTyta99bfLkyadOner+/BoaGhrmSL8HkNIIId3d3a2trb/+9a8Nw/j973/f2tra09OjWzXxvuLs7rvvXrZsmZnt3r17r7rqqieffLKtra2hoSGJPs7i+O1t1tIGBwdbr1zXXHNNfX19a2vryZMndeuRPR9I19asWZOVlbVly5bPD1a342dp7c7NGRS+eMgw5nt6empqavLz8wsKCu677z4KuvlXTl599VX6NDDmtcLTp08zXPu9DCntsxztf/bkueee8zt3h/iOX3E2Y8aMhQsXUutNmzaVlpZmZWXdeOONSfQBZmZpJhutDJwxYwatWucBszTzU9nW0hoaGuAVofDBsUJLRAARSBEEUPhSpJFYBiKACMARQOGDY4WWiAAikCIIoPClSCOxDEQAEYAjgMIHxwotEQFEIEUQQOFLkUZiGYgAIgBHAIUPjhVaIgKIQIoggMKXIo3EMhABRACOAAofHCu0RAQQgRRBAIUvRRqJZSACiAAcARQ+OFZoKQCBhQsX3nHHHQIcxXER80+y3P/oRLw/7NDf37948eKxY8fm5eXNmzfv7NmzNFpnZ+fs2bNzcnICgUB9fb3m36dN08ZBDAIofDGA4EO5CEgSvrfffnvjxo2f/ftoU/jeeust8/t4Xf7ohMsfdqirqxs/fnxzc3NLS0tFRcX06dNNUIaGhsrKymbOnNna2hoKhYqLi5cvXy4XL/QuBwEUPjm4otcrCGzevLmsrGzUqFFjx46trKysr6+3/qty86sc3nvvvTvvvLOwsLCoqGjOnDn0SxxMifzHf/zH4uLi0aNH/+AHPxgcHDRxjXH7xz/+saen5/7777/zzjvLy8tXrVpVVVV1/PhxaxPs38sU7w879PX1ZWZmbt682dze1tZmGIb53UehUGjkyJH0BnD9+vUFBQU0K2s4HGuOAAqf5g1K4vQ+/PDDq666au3atadPnz5y5MhTTz11/vz5+fPnV1dXm18lNDg4eOHCha985SuLFi06cuTIiRMnFixYMGnSJFNKFi5cmJ+ff9dddx07duzll18OBALmF/c7ujVhamxsNAxjwYIFdtRihM/l2y6bm5tj/nxSSUnJ2rVrCSGrVq0qLy+nzjs6OgzDOHjwIJ3BQbIggMKXLJ1KvjzfeustwzDOnDljTT3mqe6///u/T5o06fLly6bN4OBgTk7Orl27CCELFy4cO3Ys/QLX9evX5+fnX7p0ydHtJ5988sADD9A7vurq6vb2dmvcGOFz+er2DRs2ZGVlWfdOmTJl6dKlhJDa2tqqqiq6FI1GDcMIhUJ0BgfJggAKX7J0KvnyHBoaqqysHD169Pe+972nn376k08+MeXM+uZGfX19RkZGnuUaMWKE+WebFi5ceOutt9KyDx06ZMqoo9v29nbzDwbR1/i2bdtG9xJCUPisaOAYhQ85IBGBy5cv79mz57HHHps8eXIgEOjo6Ii546urq5s6derJ4VdfX58pkY7CRwixu6U1xLyrS+djhA+f6lJk0nOAwpeefVdd9dDQ0HXXXfcv//IvtbW11j9w8fTTTxcVFUUiEXtC5lPdP/3pT+bSv/7rv5pPda2W1K110nEcI3wuf9jBfHNjy5Ytpp/29vaYNzfOnTtnLjU2NhYUFAwMDDhGxEmdEUDh07k7yZ3b/v37V69e/eabb3Z2dm7atCkrKysUCq1evbqkpKS9vf3jjz++cOFCNBqdOHHit771rd///vcdHR2vvvrqQw891NXVZd7x5efn19TUHD9+/JVXXrn66qvNv5Lh6DYeUi5/dCLeH3YghNTV1ZWUlOzevbulpWXalcv0b36cpaqq6tChQzt37gwEAvhxlnjIaz6Pwqd5g5I4vRMnTtx2222BQCA7O7u0tPSXv/wlIeSjjz76m7/5m/z8fMMwzI+zdHd333PPPcXFxdnZ2V/+8pdra2vNG0DzSfFjjz32F3/xF/n5+bW1tea9laPbeDC5/9EJxz/sQAgxP8BcVFSUm5s7d+7c7u5u6v/MmTOzZs3KyckpLi5esmQJfoCZIpNcAxS+5OpXGmUb82pgGlWOpcpHAIVPPsYYISEEUPgSgg03gRBA4QPBhEbqEUDhU495+kRE4UufXmOliAAi8D8IoPAhFRABRCDtEEDhS7uWY8GIACKAwoccQAQQgbRDAIUv7VqOBSMCiAAKH3IAEUAE0g4BFL60azkWjAggAih8yAFEABFIOwRQ+NKu5VgwIoAI/D/6fKX8B0qduAAAAABJRU5ErkJggg=="
    },
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbEAAAFMCAIAAADKvhUhAAAAAXNSR0IArs4c6QAAIABJREFUeAHtnQ9UlNeZ/y9ShgziIGRm0ahwggKnCxbtia4mVZqDASRWV7fRYo9l4x4MidvaKhAwItv8qrXbLHv2WLX0JNLdTdCNHjNqGDGnSNxoNBsiSPinIkimCWpHhcVR/un9pb7r69s7zGXmFWTu5fseT3vvfd73nft8v8/98M68b2YIxQYFoAAUgAL3FSD3G/h/KAAFoAAUoGAiigAKQAEo8EABMPGBFmhBASgABcBE1AAUgAJQ4IECYOIDLdCCAlAACoCJqAEoAAWgwAMFwMQHWqAFBaAAFJCZiXfu3LHb7R0dHZ3YoAAUgAL3FOjo6LDb7Xfu3HFHf5mZaLfbCTYoAAWggIsCdrt9NDKxo6ODEGK32z38A+lwOEpLSx0Oh4f7i77bqMoXyYperu7m762zyqVSR0fHaGRiZ2cnIaSzs9Nd8sx4b2+v1Wrt7e1lxmXtjqp8kSzKWFFgUCzI/N550OSZKhlVy4ZSOqryRbJMtUvT9dbZQbEAJj6oDW/FfXCkmK1RlS+SFbNIB5+1t86CiXjv7LaqvC0mtycSIYBkRXBJzxy9dRZMBBPd1pm3xeT2RCIEkKwILumZo7fOgolgots687aY3J5IhACSFcElPXP01lkwEUx0W2feFpPbE4kQQLIiuKRnjt46CyaCiW7rzNticnsiEQJIVgSX9MzRW2fBRDDRbZ15W0xuTyRCAMmK4JKeOXrrLJgIJrqtM2+Lye2JRAggWRFc0jNHb50FE8FEt3XmbTG5PZEIASQrgkt65uits2Ci10yM2Xg48tX3tf/0GCXCMd4Wkwg5uZ0jknUrjeABb50FE8FEtyXvbTG5PZEIASQrgkt65uitsyPMxJ07d06fPn3cvW3OnDk2m01JOjExUfvlPS+99JIqRltbW1pamtFotFgs2dnZfX19aqiysnLmzJkGg2Hq1KklJSXquLvGoMkzByri4jqRkUWOrrcrR+iskSzHvkGxMLz/vfOhQ4fKysrOnz9/7ty5jRs3BgQE1NXVUUoTExMzMzPb72/qV9f09/fHx8cvWLCgurraZrOZzeb8/HwlvZaWlqCgoPXr1zc0NGzfvt3f37+8vJyTOaV00OSZw8FERhCZusCETG5qc/HW2UGxMLxM1E6dUhoaGvrmm28qTFy3bh0TpZTabLYxY8ZcvnxZCe3atctkMvX09FBKc3Nz4+Li1ENWrFiRkpKidgdsDJo8cxSYyAgiU9fblSN07kiWY9+gWHhETOzv79+zZ4/BYKivr1eYaDabH3/88bi4uLy8PKfTqeRQUFCQkJCg5tPS0kIIOXPmDKV03rx5Wozu3r3bZDKpe6qN7u5u9bsnlS+PdDgcvZ5tTqfTarVO33QoZuNh7T/PjhZvLyVfp9Mp3tS9nzGS9V4zMY7w1lmHw8H/WtVhZ2Jtbe3YsWP9/f1DQkLKysoUeBUXF5eXl9fW1r799tuTJk1aunSpMp6ZmZmcnKwCzul0EkKUTyGjo6O3bt2qhsrKygght27dUkeURmFhofaTSkJIaWmpFRsUgAJQ4J4CpaWlI8zEnp6eCxcuVFVV5eXlmc1m5TpRC7KKigpCSHNzM6X04ZmI60TP/7h7+wfW8zP74J5I1gdNGZIpeevsyF8navGXlJS0Zs0a7Qil9ObNm4QQ5YbJw7931p580A8OtDur3zuN+86MLHJ08RGbHD66ZuGts4NiYdjfO2tzePbZZzMyMrQjlNITJ04QQs6ePaveY7ly5YqyT3Fxsclk6u7uVu6xxMfHq8emp6fjHouqhr6Gt8Wk71V85Cgk6yNGDPk0vHV2hJmYl5d3/Pjx1tbW2travLw8Pz+/Dz74oLm5+fXXX6+qqmptbT148GBUVNT8+fMVpZRncZKTk2tqasrLyy0WC/MsTk5OTmNj444dO/AszsPXlrfF9PCvOIJnQLIjKP6wvrS3zo4wE1evXh0ZGWkwGCwWS1JS0gcffEAp/eKLL+bPnx8WFhYYGDht2rScnBz1+URK6aVLlxYuXGg0Gs1m84YNG5hntmfMmGEwGKKiovDM9sPXmbfF9PCvOIJnQLIjKP6wvrS3zo4wE4dVi0FPPmjyzBkUcfF5IiOLHF1vV47QWSNZjn2DYuGRfp7ImehwhAZNnnlRMJERRKYuMCGTm9pcvHV2UCyAiQ/kBRMfaCFdy9uVI7QASJZjH5iI78VxWx5YOW6lETwAZzkGgolgotvywMpxK43gATjLMRBMBBPdlgdWjltpBA/AWY6BYCKY6LY8sHLcSiN4AM5yDAQTwUS35YGV41YawQNwlmMgmAgmui0PrBy30ggegLMcA8FEMNFteWDluJVG8ACc5RgIJoKJbssDK8etNIIH4CzHQDARTHRbHlg5bqURPABnOQaCiWCi2/LAynErjeABOMsxEEwEE92WB1aOW2kED8BZjoFgIpjotjywctxKI3gAznIMBBPBRLflgZXjVhrBA3CWYyCYCCa6LQ+sHLfSCB6AsxwDwUQw0W15YOW4lUbwAJzlGAgmgoluywMrx600ggfgLMdAMBFMdFseWDlupRE8AGc5BoKJYKLb8sDKcSuN4AE4yzEQTAQT3ZYHVo5baQQPwFmOgWAimOi2PLBy3EojeADOcgwEE8FEt+WBleNWGsEDcJZjIJgIJrotD6wct9IIHoCzHAPBRDDRbXlg5biVRvAAnOUYOMJM3Llz5/Tp08fd2+bMmWOz2ZS53r59+5VXXgkLCxs7duyyZcsuX76s5tDW1paWlmY0Gi0WS3Z2dl9fnxqqrKycOXOmwWCYOnVqSUmJOu6uMWjyzIFKJcVsPBz56vvaf8xu0nSxcqSxkkkEzjKCaLuDYmF4f/P+0KFDZWVl58+fP3fu3MaNGwMCAurq6iilWVlZU6ZMqaioqKqqmjNnztNPP61Mur+/Pz4+fsGCBdXV1TabzWw25+fnK6GWlpagoKD169c3NDRs377d39+/vLxcm6pre9DkmUPAREYQmbrAhExuanPx1tlBsTC8TNROnVIaGhr65ptvdnR0BAQE7Nu3T4k2NjYSQk6dOkUptdlsY8aMUS8bd+3aZTKZenp6KKW5ublxcXHqCVesWJGSkqJ2B2wMmjxzFJjICCJT19uVI3TuSJZj36BYeERM7O/v37Nnj8FgqK+vr6ioIITcuHFDnXdERERRURGltKCgICEhQR1vaWkhhJw5c4ZSOm/evHXr1qmh3bt3m0wmtas2uru7O+9vdrudEOJwOHo925xOp9Vqnb7pUMzGw9p/nh0t3l5Kvk6nU7ypez9jJOu9ZmIc4a2zDoeDEN5thmFnYm1t7dixY/39/UNCQsrKyiil77zzjsFgUClGKZ01a1Zubi6lNDMzMzk5WQ05nU5CiPIpZHR09NatW9VQWVkZIeTWrVvqiNIoLCwkf7mVlpZasUEBKAAF7ilQWlo6wkzs6em5cOFCVVVVXl6e2Wyur68fVibiOtHzP+7e/oH1/Mw+uCeS9UFThmRK3jo78teJ2uu4pKSkNWvWDOt7Z+3LDfrBgXZnSik+T2QEkamLj9hkclObi7fODoqFYX/vrJ39s88+m5GRodxj2b9/vxJqampi7rFcuXJFCRUXF5tMpu7ubuUeS3x8vHq29PR03GNR1dDX8LaY9L2KjxyFZH3EiCGfhrfOjjAT8/Lyjh8/3traWltbm5eX5+fn98EHHyjP4kRERBw7dqyqqmruvU1RSnkWJzk5uaampry83GKxMM/i5OTkNDY27tixA8/iPHxteVtMD/+KI3gGJDuC4g/rS3vr7AgzcfXq1ZGRkQaDwWKxJCUlKUCklCrPbIeGhgYFBS1durS9vV1V7dKlSwsXLjQajWazecOGDcwz2zNmzDAYDFFRUXhmW1VMd8PbYtL9Qr5wIJL1BReGYw7eOjvCTBwOCTw/56DJM6dSxMV/x8LIIkfX25UjdNZIlmPfoFh4pJ8nciY6HKFBk2deFExkBJGpC0zI5KY2F2+dHRQLYOIDecHEB1pI1/J25QgtAJLl2Acm8h5YZ4QDExlBZOoCE9K4qf1+lshX34/ZeNhqtfb29nqYIJgIJrotFWDCrTSCB+R2FkzUX56D/kFgTo3rREYQmbpyY4JxSu5kwUTGbi+6YCJfLLlXDpM7kmUEEbcLJur3DkzkawdM8PURNyq3s2Ci/soEE/nayb1ymNyRLCOIuF0wUb93YCJfO2CCr4+4UbmdBRP1VyaYyNdO7pXD5I5kGUHE7YKJ+r0DE/naARN8fcSNyu0smKi/MsFEvnZyrxwmdyTLCCJuF0zU7x2YyNcOmODrI25UbmfBRP2VCSbytZN75TC5I1lGEHG7YKJ+78BEvnbABF8fcaNyOwsm6q9MMJGvndwrh8kdyTKCiNsFE/V7BybytQMm+PqIG5XbWTBRf2WCiXzt5F45TO5IlhFE3C6YqN87MJGvHTDB10fcqNzOgon6KxNM5Gsn98phckeyjCDidsFE/d6BiXztgAm+PuJG5XYWTNRfmWAiXzu5Vw6TO5JlBBG3Cybq9w5M5GsHTPD1ETcqt7Ngov7KBBP52sm9cpjckSwjiLhdMFG/d2AiXztggq+PuFG5nRWbiVu3bn3qqaeCg4MtFsuSJUuamprUOktMTCSa7aWXXlJDbW1taWlpRqPRYrFkZ2f39fWpocrKypkzZxoMhqlTp5aUlKjjAzbAxAFlUQflXjlqmkoDyTKCiNsVm4kpKSklJSV1dXU1NTVpaWkRERE3b95UzEhMTMzMzGy/v3V2dirj/f398fHxCxYsqK6uttlsZrM5Pz9fCbW0tAQFBa1fv76hoWH79u3+/v7l5eUca8FEjjiUUmCCr4+4UbmdFZuJ2qq6evUqIeT48ePKYGJi4rp167Q7KG2bzTZmzJjLly8r3V27dplMpp6eHkppbm5uXFycesiKFStSUlLUrtLo7u7uvL/Z7XZCiMPh6PVsczqdVqt1+qZDMRsPa/95drR4eyn5Op1O8abu/YyRrPea+egR2rUZs/Hw9E2HrFar52XscDgI4f3sO2GYMnzdCxcuEEI+//xz5SUSExPNZvPjjz8eFxeXl5fndDqV8YKCgoSEBHUaLS0thJAzZ85QSufNm6fF6O7du00mk7qn0igsLNS8I/9zs7S01IoNCkABKHBPgdLSUp9g4p07d55//vlnnnlGRVhxcXF5eXltbe3bb789adKkpUuXKqHMzMzk5GR1N6fTSQix2WyU0ujo6K1bt6qhsrIyQsitW7fUEUoprhM9/+OOSyfPtRJrT7mdleQ6MSsrKzIy0m63a/mltisqKgghzc3NlNKHZKJ6TkopPk/UquHalvtTJyZfJMsIIm5Xhs8T165dO3ny5JaWFnc23Lx5kxCi3DB5yPfO2pcAE7VquLaBCVdN5BiR21mxmXj37t21a9c+8cQT58+f51TbiRMnCCFnz56llCr3WK5cuaLsX1xcbDKZuru7lXss8fHx6nnS09Nd77GoUVwnaqUYsC33ymFSRrKMIOJ2xWbiyy+/HBIS8uGHH95/5KZd+fivubn59ddfr6qqam1tPXjwYFRU1Pz58xWTlGdxkpOTa2pqysvLLRYL8yxOTk5OY2Pjjh078CzOQ5Y1MPGQAvrs4XI7KzYTmVvAhBDlQesvvvhi/vz5YWFhgYGB06ZNy8nJUZ9PpJReunRp4cKFRqPRbDZv2LCBeWZ7xowZBoMhKioKz2w/5JqUe+Uw4iBZRhBxu2IzcWR1x+eJfP2BCb4+4kbldhZM1F+ZYCJfO7lXDpM7kmUEEbcLJur3DkzkawdM8PURNyq3s2Ci/soEE/nayb1ymNyRLCOIuF0wUb93YCJfO2CCr4+4UbmdBRP1VyaYyNdO7pXD5I5kGUHE7YKJ+r0DE/naARN8fcSNyu0smKi/MsFEvnZyrxwmdyTLCCJuF0zU7x2YyNcOmODrI25UbmfBRP2VCSbytZN75TC5I1lGEHG7YKJ+78BEvnbABF8fcaNyOwsm6q9MMJGvndwrh8kdyTKCiNsFE/V7BybytQMm+PqIG5XbWTBRf2WCiXzt5F45TO5IlhFE3C6YqN87MJGvHTDB10fcqNzOgon6KxNM5Gsn98phckeyjCDidsFE/d6BiXztgAm+PuJG5XYWTNRfmWAiXzu5Vw6TO5JlBBG3Cybq9w5M5GsHTPD1ETcqt7Ngov7KBBP52sm9cpjckSwjiLhdMFG/d2AiXztggq+PuFG5nQUT9VcmmMjXTu6Vw+SOZBlBxO2Cifq9AxP52gETfH3EjcrtLJiovzLBRL52cq8cJnckywgibhdM1O8dmMjXDpjg6yNuVG5nwUT9lQkm8rWTe+UwuSNZRhBxu2IzcevWrU899VRwcLDFYlmyZElTU5PqxO3bt1955ZWwsLCxY8cuW7bs8uXLaqitrS0tLc1oNFosluzs7L6+PjVUWVk5c+ZMg8EwderUkpISdXzABpg4oCzqIDChSiFZQ25nxWZiSkpKSUlJXV1dTU1NWlpaRETEzZs3lfrLysqaMmVKRUVFVVXVnDlznn76aWW8v78/Pj5+wYIF1dXVNpvNbDbn5+croZaWlqCgoPXr1zc0NGzfvt3f37+8vJxTzWAiRxxKqdwrh8kdyTKCiNsVm4la3a9evUoIOX78OKW0o6MjICBg3759yg6NjY2EkFOnTlFKbTbbmDFj1MvGXbt2mUymnp4eSmlubm5cXJx6zhUrVqSkpKhdpdHd3d15f7Pb7YQQh8PR69nmdDqtVuv0TYdiNh7W/vPsaPH2UvJ1Op3iTd37GSNZ7zXz0SO0azNm4+Hpmw5ZrVbPy9jhcBBCOjs7GXSoXaK2+I3f//7377//vrJPTk5OSEjI3LlzL126xD9KG71w4QIh5PPPP6eUVlRUEEJu3Lih7hAREVFUVEQpLSgoSEhIUMdbWloIIWfOnPn6qHnz5q1bt04N7d6922QyqV2lUVhYSP5yKy0ttWKDAlAACtxToLS0dGiYGBMTU1FRQSn9+OOPg4KCiouLv/e97y1dupRBkrvunTt3nn/++WeeeUbZ4Z133jEYDNqdZ82alZubSynNzMxMTk5WQ06nkxBis9kopdHR0Vu3blVDZWVlhJBbt26pI5RSXCd6/scdl06eayXWnnI76yvXiUajsa2tTXkDu2rVKkppXV2d2WzW8ojTzsrKioyMtNvtyj7Dx0TtHPB5olYN1zY+YnPVRI4RuZ31lc8TLRaL8gZ2xowZ//Ef/0EpbW5uHjt2rCc1tHbt2smTJ7e0tKg7D997Z/UlKKVgolYN17bcK4fJF8kygojb9RUmrly58tvf/vY//MM/BAUFORwOSunBgwe1dzwGlPju3btr16594oknzp8/r91Buceyf/9+ZbCpqYm5x3LlyhUlVFxcbDKZuru7lUvU+Ph49Tzp6emu91jUKJiolWLANjAxoCwSDMrtrK8w8caNG2vXrl28ePGRI0eUotm8efMvfvELfgG9/PLLISEhH374Yfv9Tf34LysrKyIi4tixY1VVVXPvbcqplGdxkpOTa2pqysvLLRYL8yxOTk5OY2Pjjh078CwOX/xBo3KvHCZ9JMsIIm7XV5ioT8G/vAP85576oLXyzHZoaGhQUNDSpUvb29vVl7h06dLChQuNRqPZbN6wYQPzzPaMGTMMBkNUVJR6KvVApoH3zowgTBeYYASRpiu3s77CxCNHjnz00UdK0fzmN79JSEhIT0+/fv26L5cRmMh3R+6Vw+SOZBlBxO36ChPj4+PLysoopbW1tYGBgfn5+XPmzPn7v/97X1YWTOS7A0zw9RE3KrezvsLEsWPHtra2fl0lhYWFf/d3f0cp/eyzz8LDw325bsBEvjtyrxwmdyTLCCJu11eYGBoaWl9fTyl95plniouLKaWtra1Go9GXlQUT+e4AE3x9xI3K7ayvMPF73/teSkrK66+/HhAQ8Mc//pFSevTo0ejoaF+uGzCR747cK4fJHckygojb9RUmtrW1Pf/889/61rfefPNNRc2f/vSnP/7xj31ZWTCR7w4wwddH3KjczvoKE0WsDzCR75rcK4fJHckygojb9SEmfv009f79+//fve3AgQP9/f0+LiuYyDcImODrI25Ubmd9hYkXLlyIjo4OCgqaeW8LCgqKjY1tbm725boBE/nuyL1ymNyRLCOIuF1fYeLChQtTU1OvXbumSOlwOFJTU9PS0nxZWTCR7w4wwddH3KjczvoKE4OCgmpra7VVUlNT4+H34miPepRtMJGvttwrh8kdyTKCiNv1FSaGhoaePHlSq+OJEydCQ0O1I77WBhP5jgATfH3EjcrtrK8wcdWqVXFxcadPn757bzt16lR8fHxGRoYv1w2YyHdH7pXD5I5kGUHE7foKE2/cuLF48WI/Pz/Dvc3Pz+9v//ZvtT+o4oMSg4l8U4AJvj7iRuV21leYqNTHhQsXDt3bLly44PsVAybyPZJ75TC5I1lGEHG7I8zEnw22+bKyYCLfHWCCr4+4UbmdHWEmfpe7Pfvss75cN2Ai3x25Vw6TO5JlBBG3O8JMFFc4/B7LoN4BE4NKJOgOcjsLJuovS1wn8rWTe+UwuSNZRhBxu2Cifu/ARL52wARfH3GjcjsLJuqvTDCRr53cK4fJHckygojbBRP1ewcm8rUDJvj6iBuV21kwUX9lgol87eReOUzuSJYRRNwumKjfOzCRrx0wwddH3KjczoKJ+isTTORrJ/fKYXJHsowg4nbFZuLx48cXLVo0ceJEQsh7772n2pCRkUE0W0pKihq6du3aypUrx40bFxISsnr16q6uLjV09uzZ73znO4GBgZMnT/7Vr36ljrtrgInulFHGgQm+PuJG5XZWbCbabLbXXnvtwIEDrkxMTU1tv79dv35drb/U1NSEhITTp09/9NFH06ZNS09PV0KdnZ3h4eE//OEP6+rq9uzZYzQalZ9UVQ90bYCJrppoR+ReOdpMKaVIlhFE3K7YTFR1d2XikiVL1KjaaGhoIIR8+umnysiRI0f8/Py+/PJLSunOnTtDQ0N7enqU0KuvvhobG6seOGADTBxQFnUQmFClkKwht7PSMjEkJMRiscTExGRlZTkcDqUo33rrrfHjx6sF2tfX5+/vf+DAAUrpqlWrtBg9duwYIUR7gakc1d3d3Xl/s9vthBCHw9Hr2eZ0Oq1W6/RNh2I2Htb+8+xo8fZS8nU6neJN3fsZI1nvNfPRI7RrM2bj4embDlmtVs/L2OFwEEI6OztVzjANwvSHqctcJ+7Zs+fgwYO1tbXvvffeN7/5zVmzZim/Arhly5aYmBjtHCwWy86dOymlzz333Jo1a9RQfX09IaShoUEdURqFhYWaDyr/3CwtLbVigwJQAArcU6C0tNQXmagF2cWLFwkhf/jDHyilD89EXCd6/scdl06eayXWnnI7K+d1opaJlFKz2fzb3/6WUvrw7521Z8bniVo1XNtyf+rE5ItkGUHE7cr5eaLWD7vd7ufnd/DgQUqpco+lqqpK2eHo0aPMPZbe3l4llJ+fj3ssWhl1tIEJHaIJcYjczorNxK6urup7GyGkqKiourq6ra2tq6srOzv71KlTra2tf/jDH7797W9HR0d3d3cr1Zaamjpz5sxPPvnkxIkT0dHR6rM4HR0d4eHhq1atqqur27t3b1BQEJ7Fecj1KffKYcRBsowg4nbFZmJlZSVzxyMjI+PWrVvJyckWiyUgICAyMjIzM/Py5cuqQ9euXUtPTw8ODjaZTC+++OKAz2xPmjRp27Zt6iHuGnjv7E4ZZRyY4OsjblRuZ8Vm4shWFZjI11/ulcPkjmQZQcTtgon6vQMT+doBE3x9xI3K7SyYqL8ywUS+dnKvHCZ3JMsIIm4XTNTvHZjI1w6Y4OsjblRuZ8FE/ZUJJvK1k3vlMLkjWUYQcbtgon7vwES+dsAEXx9xo3I7Cybqr0wwka+d3CuHyR3JMoKI2wUT9XsHJvK1Ayb4+ogbldtZMFF/ZYKJfO3kXjlM7kiWEUTcLpio3zswka8dMMHXR9yo3M6CiforE0zkayf3ymFyR7KMIOJ2wUT93oGJfO2ACb4+4kbldhZM1F+ZYCJfO7lXDpM7kmUEEbcLJur3DkzkawdM8PURNyq3s2Ci/soEE/nayb1ymNyRLCOIuF0wUb93YCJfO2CCr4+4UbmdBRP1VyaYyNdO7pXD5I5kGUHE7YKJ+r0DE/naARN8fcSNyu0smKi/MsFEvnZyrxwmdyTLCCJuF0zU7x2YyNcOmODrI25UbmfBRP2VCSbytZN75TC5I1lGEHG7YKJ+78BEvnbABF8fcaNyOwsm6q9MMJGvndwrh8kdyTKCiNsFE/V7BybytQMm+PqIG5XbWTBRf2WCiXzt5F45TO5IlhFE3C6YqN87MJGvHTDB10fcqNzOis3E48ePL1q0aOLEiYSQ9957Ty2yu3fvFhQUTJgw4bHHHktKSjp//rwaunbt2sqVK8eNGxcSErJ69equri41dPbs2e985zuBgYGTJ0/+1a9+pY67a4CJ7pRRxuVeOUzuSJYRRNyu2Ey02WyvvfbagQMHGCZu27YtJCTEarWePXt28eLFTz755O3btxWTUlNTExISTp8+/dFHH02bNi09PV0Z7+zsDA8P/+EPf1hXV7dnzx6j0VhcXMz3FUzk6wNM8PURNyq3s2IzUa0qLRPv3r07YcKEX//610q0o6MjMDBwz549lNKGhgZCyKeffqqEjhw54ufn9+WXX1JKd+7cGRoa2tPTo4ReffXV2NhY9fwDNsDEAWVRB+VeOWqaSgPJMoKI25WQiRcvXiSEVFdXq67Mnz//Jz/5CaX0rbfeGj9+vDre19fn7+9/4MABSumqVauWLFmiho4dO0YIuX79ujqiNLq7uzvvb3a7nRDicDh6PducTqfVap2+6VDMxsPaf54dLd5eSr5Op1O8qXs/YyTrvWY+eoR2bcZsPDx90yEAP5ciAAAV+ElEQVSr1ep5GTscDkJIZ2cngw61S9TWsDa014knT54khHz11VfqK77wwgvLly+nlG7ZsiUmJkYdp5RaLJadO3dSSp977rk1a9aoofr6ekJIQ0ODOqI0CgsLyV9upaWlVmxQAApAgXsKlJaWji4m4jrR8z/uuHTyXCux9pTbWQmvE4f1vbP2shGfJ2rVcG3jIzZXTeQYkdtZCT9PVO6xvPHGG0r9dXZ2MvdYqqqqlNDRo0eZeyy9vb1KKD8/H/dYHnIBy71yGHGQLCOIuF2xmdjV1VV9byOEFBUVVVdXt7W1UUq3bds2fvz4gwcP1tbWLlmyhHkWZ+bMmZ988smJEyeio6PVZ3E6OjrCw8NXrVpVV1e3d+/eoKAgPIvzkGUNTDykgD57uNzOis3EysrKv7zhQTIyMiilyjPb4eHhgYGBSUlJ586dU8vr2rVr6enpwcHBJpPpxRdfHPCZ7UmTJm3btk09xF0D753dKaOMy71ymNyRLCOIuF2xmTiyuoOJfP2BCb4+4kbldhZM1F+ZYCJfO7lXDpM7kmUEEbcLJur3DkzkawdM8PURNyq3s2Ci/soEE/nayb1ymNyRLCOIuF0wUb93YCJfO2CCr4+4UbmdBRP1VyaYyNdO7pXD5I5kGUHE7YKJ+r0DE/naARN8fcSNyu0smKi/MsFEvnZyrxwmdyTLCCJuF0zU7x2YyNcOmODrI25UbmfBRP2VCSbytZN75TC5I1lGEHG7YKJ+78BEvnbABF8fcaNyOwsm6q9MMJGvndwrh8kdyTKCiNsFE/V7BybytQMm+PqIG5XbWTBRf2WCiXzt5F45TO5IlhFE3C6YqN87MJGvHTDB10fcqNzOgon6KxNM5Gsn98phckeyjCDidsFE/d6BiXztgAm+PuJG5XYWTNRfmWAiXzu5Vw6TO5JlBBG3Cybq9w5M5GsHTPD1ETcqt7Ngov7KBBP52sm9cpjckSwjiLhdMFG/d2AiXztggq+PuFG5nQUT9VcmmMjXTu6Vw+SOZBlBxO2Cifq9AxP52gETfH3EjcrtLJiovzLBRL52cq8cJnckywgibhdM1O8dmMjXDpjg6yNuVG5nwUT9lQkm8rWTe+UwuSNZRhBxu3IysbCwkGi22NhYxaHbt2+/8sorYWFhY8eOXbZs2eXLl1Xn2tra0tLSjEajxWLJzs7u6+tTQ+4aYKI7ZZRxYIKvj7hRuZ2VlolxcXHt97c//elPSv1lZWVNmTKloqKiqqpqzpw5Tz/9tDLe398fHx+/YMGC6upqm81mNpvz8/MHLVkwkS+R3CuHyR3JMoKI25WWiQkJCYwrHR0dAQEB+/btU8YbGxsJIadOnaKU2my2MWPGqJeNu3btMplMPT09zBmYLpjICMJ0gQlGEGm6cjsrLRODgoImTpz45JNPrly5sq2tjVJaUVFBCLlx44ZamhEREUVFRZTSgoICLUNbWloIIWfOnFH3VBvd3d2d9ze73U4IcTgcvZ5tTqfTarVO33QoZuNh7T/PjhZvLyVfp9Mp3tS9nzGS9V4zHz1CuzZjNh6evumQ1Wr1vIwdDgchpLOzU4UG0yBM/9F0bTbbu+++e/bs2fLy8rlz50ZERPzv//7vO++8YzAYtBOYNWtWbm4upTQzMzM5OVkNOZ1OQojNZlNH1AbzSSUhpLS01IoNCkABKHBPgdLSUl9kooowSumNGzdMJtObb745JEzEdaLnf9xx6eS5VmLtKbezcl4naplIKX3qqafy8vKG5L2z9sz4PFGrhmtb7k+dmHyRLCOIuF05P0/U+tHV1RUaGvpv//Zvyj2W/fv3K9GmpibmHsuVK1eUUHFxsclk6u7u1p7HtQ0mumqiHQEmtGrI1JbbWTmZuGHDhg8//LC1tfXkyZMLFiwwm81Xr16llGZlZUVERBw7dqyqqmruvU2pVOVZnOTk5JqamvLycovFgmdxHn4Ny71yGH2QLCOIuF05mbhixYqJEycaDIZJkyatWLGiublZcUh5Zjs0NDQoKGjp0qXt7e2qc5cuXVq4cKHRaDSbzRs2bMAz26oyuhvAhG7pfPxAuZ2Vk4mPpqTw3pmvs9wrh8kdyTKCiNsFE/V7BybytQMm+PqIG5XbWTBRf2WCiXzt5F45TO5IlhFE3C6YqN87MJGvHTDB10fcqNzOgon6KxNM5Gsn98phckeyjCDidsFE/d6BiXztgAm+PuJG5XYWTNRfmWAiXzu5Vw6TO5JlBBG3Cybq9w5M5GsHTPD1ETcqt7Ngov7KBBP52sm9cpjckSwjiLhdMFG/d0PFRMaDyFff1z8nXzoSmPAlN4ZyLnI7y6zHmI2HrVZrb2+vhwoOioWR+f5ED2f/kLsNmjxzfqWSYjYeZkR37TIHCtqVe+UwpiBZRhBxu8x6BBO9sBJM5IsFTPD1ETcqk7MMAV27YKIXhQom8sWSaeXwM6WUItlBJfLNHVwhyIyAiV4YBybyxQIm+PqIG5XJWYaArl0w0YtCBRP5Ysm0cviZ4jpxUH18dgdXCDIjYKIX3oGJfLHARL4+4kZlcpYhoGsXTPSiUIePia7GeDEtn9lVppUzqKhIdlCJfHMH17XGjICJXhgHJvLFAib4+ogblclZhoCuXTDRi0IFE/liybRy+Jni88RB9fHZHVwhyIyAiV54BybyxQIT+fqIGxXUWQZ2HnbBRC8KFUzkiyXoyuEn5S6KZN0p4zvjHkKQ2Q1M9MJBMJEvFjDB10fcqKDOMrDzsAsmelGoj5KJA/rnOlfX3Vz3eWQjgq4cffogWX26Dd9RrmtB3wiY6IVHI85ETzz2Ip+h3hWYGGpFfeV8PuisJ2tB3z5gohdlBybyxfLBlcOf8MNEkezDqMc51hVkA+7suttQjYCJAwo+8KAQTPSwMgbO0GXU9WwuuzwYACYeaCFXa1BnXevEdcRDSVwPfMQjYCL9zW9+ExkZGRgYOHv27E8++YTjnExM1F1nHH20K2c4zs956Ucf0iarvrpr1mrI1xquU+V8t7E22QEPlGlwtDNx7969BoNh9+7d9fX1mZmZ48ePv3LlirvyBRP5pa8Ukyffocs/j76oq2uu53Hdh1LqyW6u+wxtsq4Tc33FERwZ2mRHMBFPXnq0M3H27Nlr165VKvLOnTtPPPHEL3/5S9cCVUbARH5JjcKVM1J/APhGDHl0FDo7Sn97oKenx9/f/7333lMh+KMf/Wjx4sVql1La3d3deX/74osvCCGtra0Oz7b29vbS0tK/ztk7dcN/jYZ/f52zd/Tki2RlLWnF2fb2ds9WuaO1tZUQ0tHRoeWGti3S77F8+eWXhJCPP/5YTSAnJ2f27Nlq9+tGYWEhwQYFoAAU4Cpgt9u13NC2ZWOi9jrxxo0bFy9e7OjouH/hOMj/2+12Qojdbh9kP1nCoypfJCtL2bJ5eOtsR0eH3W6/c+eOloPatkhM9OS9szY3b9vefv7o7fl9bf9RlS+S9bXyG6r5DLmzIjGRUjp79ux//Md/VNS8c+fOpEmTOPdYvBV9yMX1dgKPeP9RlS+SfcTV9chebsidFYyJe/fuDQwM/P3vf9/Q0LBmzZrx48dfvnx5qNQfcnGHamLDdJ5RlS+SHaYqGvHTDrmzgjGRUrp9+/aIiAiDwTB79uzTp08PoSXd3d2FhYVf/+8QntOXTzWq8kWyvlyKDzO3IXdWPCY+jHw4FgpAASjAVwBM5OuDKBSAAqNLATBxdPmNbKEAFOArACby9UEUCkCB0aUAmDi6/Ea2UAAK8BUAE+kvfvGLuXPnGo3GkJAQjlh3794tKCiYMGHCY489lpSUdP78ec7OPhu6du3aypUrx40bFxISsnr16q6urgGnmpiYqP0vo1566aUBd/PNQU++Te7dd9+NjY0NDAyMj48vKyvzzUQ8mdWgyZaUlGitDAwM9OS0PrjP8ePHFy1aNHHiREKI9ksPmKlWVlbOnDnTYDBMnTq1pKSEiXrSBRPp5s2bi4qK1q9fz2fitm3bQkJCrFbr2bNnFy9e/OSTT96+fdsTiX1qn9TU1ISEhNOnT3/00UfTpk1LT08fcHqJiYmZmZnt97fOzs4Bd/PBQU++Te7kyZP+/v7//M//3NDQsGnTpoCAgM8//9wHcxl0Sp4kW1JSYjKZ7jvZPoTP8w46vaHdwWazvfbaawcOHOAwsaWlJSgoaP369Q0NDdu3b/f39y8vL/d2GmDi/ylWUlLCYeLdu3cnTJjw61//Wtm7o6MjMDBwz5493so9svs3NDQQQj799FNlGkeOHPHz8/vyyy9dZ5WYmLhu3TrXcd8f8eTb5JYvX/7888+rufzN3/yNWBfC6sw9SZZf2OqpBGpwmJibmxsXF6fmsmLFipSUFLXrYQNM/D+h+KVz8eJFQkh1dbUq6/z583/yk5+oXSEab7311vjx49Wp9vX1+fv7HzhwQB1RG4mJiWaz+fHHH4+Li8vLy3M6nWrIlxse/hfxU6ZM+dd//Vc1kc2bN3/rW99Su6I0PEy2pKTE398/IiJi8uTJixcvrqurEyVBd/PkMHHevHnav+W7d+82mUzuzuNuHEz8P2X4TDx58iQh5KuvvlJ1fOGFF5YvX652hWhs2bIlJiZGO1WLxbJz507tiNIuLi4uLy+vra19++23J02atHTpUtd9fHDEk2+To5QGBASUlpaq89+xY8df/dVfqV1RGh4m+/HHH//7v/97dXX1hx9+uGjRIpPJxPmaLCFy5zAxOjp669atahZlZWWEkFu3bqkjnjTkZ+Krr76q/YxZ225sbFQ1koOJ/GQ9Z6IqC6W0oqKCENLc3Kwd9M22h5gYVUzUOtXb2zt16tRNmzZpB4Vrg4kPa9nVq1cb3Ww9PT3q2flMFOW9Mz9Zz987q7JQSm/evEkI0fFZtfYkj6bt4dvJUfXemVH++9///g9+8ANmUKwuh4l47zyUVvKZqNxjeeONN5SX7OzsFPceS1VVlZLF0aNH3d1j0Sp74sQJQsjZs2e1gz7b9uTb5JYvX75o0SI1hblz54p7j8Wrr87r7++PjY392c9+puYuYoPDxNzc3Pj4eDWp9PR03GNR1fCi0dbWVl1d/fOf/zw4OLj63qY+tRcbG6vegti2bdv48eMPHjxYW1u7ZMkScZ/FmTlz5ieffHLixIno6Gj1WZw//vGPsbGxym/DNjc3v/7661VVVa2trQcPHoyKipo/f74Xgo7oru6+TW7VqlV5eXnK1E6ePPmNb3zjjTfe+PrDk8LCQqGfxRnwq/O0yf785z8/evToxYsXP/vssx/84AePPfZYfX39iFqk88W7urqU5UkIKSoqqq6ubmtr+/onmPLy8latWqWcVHkWJycnp7GxcceOHXgWR6fWGRkZ2g8ZCSGVlZXKuQgh6mOfyjPb4eHhgYGBSUlJ586d0/l6I3rYtWvX0tPTg4ODTSbTiy++qNJf+eEeJfEvvvhi/vz5YWFhgYGB06ZNy8nJEej5RHffJpeYmJiRkaFq/+6778bExBgMhri4OKGf2R7wq/O0yf70pz9VvlsvPDw8LS3tzJkzqghiNSorK5l1qhiakZGRmJio5lJZWTljxgyDwRAVFaUuXjXqSUP+eyyeqIB9oAAUgAKKAmAiKgEKQAEo8EABMPGBFmhBASgABcBE1AAUgAJQ4IECYOIDLdCCAlAACoCJqAEoAAWgwAMFwMQHWqAFBaAAFAATUQNQAApAgQcKgIkPtEALCkABKAAmogagABSAAg8UABMfaIHWyCqQkZGxZMmS4ZuD9r8Ao5TW1dUtW7YsMjKSEKL9illlAu5+5+T27duvvPJKWFjY2LFjly1bpv0q/7a2trS0NKPRaLFYsrOz+/r6hi8XnHn4FAATh09bnNk7BYaJiefOnVN+JUJh4meffXb48GFK6f/8z/9kZ2fv2bNnwoQJDBM5v3OSlZU1ZcqUioqKqqqqOXPmPP3000qS/f398fHxCxYsqK6uttlsZrM5Pz/fu/yxt28oACb6hg+jbxb79u2Lj49/7LHHwsLCkpKSsrOztf+Fv/ptFC+88EJISEhoaOjixYtbW1sVnRR6/tM//ZPZbB43btxLL72kfhUmc9qbN29eu3ZtzZo1L7zwQkJCQkFBQXJyMvPFMJGRkQwT3f3OSUdHR0BAwL59+5RpNDY2EkJOnTpFKbXZbGPGjFEvG3ft2mUymdRZjT57Bc4YTBTYPHGn/tVXX33jG98oKipqbW2tra3dsWNHV1fX8uXLU1NTlV+Y6+np6e3t/eY3v7l69era2tqGhoaVK1fGxsYqlMnIyAgODl6xYkVdXd37779vsVg2btxIKR3wtIpKxcXFhJCVK1e6isYwkfPFtMpXjt+4cUM9SURERFFREaW0oKAgISFBHW9paSGEiPslNGoio7ABJo5C00c+5c8++4wQcunSJe1UmPfO//mf/xkbG3v37l1ln56eHqPRePToUUppRkZGWFiY+stZu3btCg4OvnPnzoCnvX79+ssvv6xeJ6ampjY1NWlfl2Ei5wcM3nnnHYPBoD121qxZubm5lNLMzMzk5GQ15HQ6CSE2m00dQUMUBcBEUZySap79/f1JSUnjxo37/ve//7vf/e769esK6bT3WLKzs/39/cdqNj8/P+UXtTIyMp599llVkZqaGoWwA562qalJ+UUq9fPEQ4cOqcdSSsFErRpog4mogZFR4O7duydOnNi8efP06dMtFktLSwtznZiVlTV79uwLf7l1dHQo9ByQiZRS19Oq6TH3ndVxhol476wqMzobYOLo9N2Hsu7v7580adK//Mu/ZGZman8m5Xe/+11oaOiAX/GtvHdWf6Pyt7/9rfLeWZuVelrt4IBtholf3zBx96Muyj2W/fv3K+dpampi7rFcuXJFCRUXF5tMpu7u7gFfEYO+rACY6MvuSDu306dPb9my5dNPP21ra3v33XcNBoPNZtuyZUtERERTU9Of/vSn3t5ep9MZHR393e9+97//+79bWloqKyt//OMfK79NrNxjSU9Pr6+vLysrCw8PV35rZcDTuhOxp6dH+X2PiRMnZmdnV1dXX7hwQdnZ3Y+6UEqzsrIiIiKOHTtWVVU1996mHKI8i5OcnFxTU1NeXm6xWPAsjjvlfXwcTPRxg+ScXkNDQ0pKisViCQwMjImJ2b59O6X06tWrzz33XHBwsPqTOO3t7T/60Y/MZnNgYGBUVFRmZqZy2ai8y968efPjjz8eHBycmZmpXJENeFp3Cio/QaN9AEj75nrA3zmhlCrPbIeGhgYFBS1durS9vV09/6VLlxYuXGg0Gs1m84YNG/DMtqqMWA0wUSy/MNs/K8B88ghRoMAQKgAmDqGYONUjUgBMfERCj8qXARNHpe2CJw0mCm6gT08fTPRpezA5KAAFHrECYOIjFhwvBwWggE8rACb6tD2YHBSAAo9YATDxEQuOl4MCUMCnFQATfdoeTA4KQIFHrACY+IgFx8tBASjg0wqAiT5tDyYHBaDAI1YATHzEguPloAAU8GkFwESftgeTgwJQ4BEr8P8BGtpUdV7yWO8AAAAASUVORK5CYII="
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![image.png](attachment:image.png)  \n",
    "然后看激活层的问题，这边激活层之前我们的W没有做上面的处理，发现数值很大，经过tanh后大部分处于图像的两端，这样会造成一个很大的问题，反向传播时效率很低甚至是不能传播，因此对隐藏层W也要优化  \n",
    "![image-2.png](attachment:image-2.png)\n"
   ]
  },
  {
   "attachments": {
    "image-2.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAAGdCAIAAAAuRy6hAAAAAXNSR0IArs4c6QAAHMNJREFUeAHtnQ10FPW5xsfDIelBjgSFU0JgN18QICAkJAvhI3AxJAiUCLcSigpaDQZDBYVg+AjLRzafO2MpeA94KcG2FkrVmHN7U9oSgVYEWs7lU0oRibjHgthTNoZiPKfL3oaB2WE2+59HrTK7PHOeY9//BzrM/Pq8739mZ0byc+MRAI6ABMzhFB4BP0EhBNARICjQYeIkgkIGoCNAUKDDxEkEhQxAR4CgQIeJkwgKGYCOwDcKis/n83g8Xq+3hZvFjoDX6/V4PD6fLxQ13ygoHo9H4mbhI+DxeCwBitfrlSQpfklZ4gqXqdJfew5RyrMVoDIfWgfKVlmOKH6NC1RimQtU7q/ngcrpNQ9RUmkFooTnVkmS5PV6v0ZQNm7caLfbo6OjHQ7HoUOHQv2X/H5/S0uLJEmJK1z91smmyvx1KaJBJQqorIdrQcW/6EaUWC2DSnbJoKb8fgGoSb0XIOq/WkGUVFohSVJLS0uo0/dVU8+OHTuioqK2bt367rvvFhYWxsTEfPzxx6H+YwTFFBeQkim/hyiZ1HsBQkn/1crXDorD4SguLlbJ8Pl8vXv3rqysJCgGpzHlQ5sQmaB8/vnnnTp1qq+v18iYM2fOtGnTtKYhoKNoQIQKIhOUjz76SJKkd955RwOipKTE4XBoTb/f39bWpq0E1VUPa5RQlCS75DsXFKfTaVgPEpQ7DhQk9dBREqvRJU/EOorf73c4HAsWLFBzjc/ni4uLYzFrqGQJSjseO3bsiI6O3rZt26lTp+bNmxcTE3Px4kV9jaKPWcwKko46FJk1igrBhg0bbDZbVFSUw+E4ePCgngxDTFDuaFAMNAiaBIWgCPAIDKmgJNQtT/rFWlPZNtcgmpjuBBW/3g3KvqEW0eiHakA5ZrtBJVXKoF4+PQbRhLHliLKzVn69l/ADFAARQTHFBaQkqVJGKHn59BiEkgljywlKwG9AO4lf70bsxL6hFrST0Q/VmPKhTSAo7YZDR9GACBUQFIIClSkEhaAQFKCMVacw9YTKOFo/HYWOQkeho5hdUNEMwzSgo9BR6Ch0FDoKzAA0kcUsUw9BEd33MeVDm8AapZ0k1VGm/ub7098uMtXwJ2REk5KWgHI84gY1vHEZoscPPQ4qrVAG1W+tAgp58ij+RTe4h7ObZlvupiBBEUADUtJvrUJQbpgNYifDn5BBO5mUtAS0E8cjbsROhjcuA//P+vihxwVkGIYIClMPlH0ICkEhKNCKh6AQFIJitvwxFCKCJlMPHYWOQkeho8AMQBN5wU2QdNQhph6mHqYeyE0ICkEhKKxRYAagiaxRWKN8AVAmDFmaO6zMXF3n5gJKe0oGNbLADSo3bRUi8G2Ug0qU8U3PgwJv9eHTRs6sRZQxfZ3l7h4TFAE0OAHgTISSkTNrCUrAb0A7GVngRuwkN20VHQXKI19iklqj0FHoKCbwEBQBIuoQmFDwaUw9N0pdsJJNe0pm6hFAwxqFNQpXPdcXz3QUcRoSuIh+iI5CR6Gj0FGAz8XobUMQ01HoKHQUOgodpf1OEHCjJ7frXBazLGZ5UxD6Yl3EgpI1cc3YydWmSi6XEeHP/6UuVkDlDlmJKEF2gzJ90FqbAL651L6hNsEtI7p/gYIodd7X/E1Bk4v2tw6rl/AJioZFcEBQ2pEhKMFkGHoICkExf9PH9LeLCApBISi3FiKCFlOPIdEEN+kodBQ6isBDbh2iowRbiKGHjkJHoaPcahuCFh3F4B/BTToKHYWOIvCQW4foKMEWYuiho9BR6Ci32oagpTpK/BpX8LfGg3uG/s8KRJmPukGN+J4b1OCGMkRN5/qDqjj5ICj7S7WgRv+2BFHqEgXRgB9Y76YgQRFAA1Jif6kWoWT0b0sQSlKXKAQl4DegnYz4nhuxk8ENZaCdNJ3rLyDDMERQAjUKHcUAh74ZmaA4nU5Jt6WkpAgKFO1nBgRFT4YhjlhQUlNTL9zcPvnkE4LSYT4y0CBoRiwoQ4cOFcOhH+WqR4CIOhSxoHTp0iU2NjYhIWH27Nnnz5/XY6HGbW1tLTc3j8cjSRJTjwCXyASlsbFx586dx44d27VrV1ZWls1m+/TTTw2sGOoYgiKgpOLkg5EJip6Jy5cv33PPPVu2bNF3+v1+OgqXxwYk/BkZGaWlpcZeXZs1ithO7ghHaW1t7d69+/r163VgGEOCcoeCsnjx4r179zY3N+/fvz8nJ6dHjx6XLl0y0qFrE5Q7FJSCgoLY2NioqKi4uLiCgoKzZ8/qqOggJCh3KCgdsCDsUkHJfGNh1m+Wmsq2qQbRhPEuUBPTnaBsW6sQgc+TJshu8FHW1MXot2z7rVXAL2Qiz8bmDlk5YVCJ5V5ITFAE3OCfYSEoN8wGsRPbphrQTiaMd4F2MjHdidiJbWsVHUWYP77CIFOPwEvUITpKO18EhaBAPkNQCApBEb1Ux5QPbQJTD1MP9L4ngkJQCAqUdggKQSEoZm/900oQ04Cph45CR6Gj0FFgBqCJvI7C1PMFQBk4v2LwIsVUQ55VENm2VIMauEwBNWpGLSLTjxhrE2yba0Ah72BW59h+XIXo4XfmIXrod49b7u4xQRFAQ1DaLUdNPQSFoJgkIIIiQEQdoqPQUaAyhaAQFIJiknACw0w9TD0BGgQRQSEoAjwCQwSFoARoEEQEhaAI8AgMERSCEqBBEBEUgiLAIzBEUAhKgAZBRFAIigCPwJAKyoTBJblDV5rr7jm5gIYVyaCQr7qqc4Y/LiNadmw6qHF5VaDSn5RB5by1CNGAFQqifkus9+ZqgiKABqQk/UkZoSTnrUUIJQNWKAQl4Dd0FAE0BIWgMPVcr2DAAmVYkUxHoaMEbEPADUEhKAQl5ApIAId+iDVKgCE6ip4MQ0xQCAqLWRazwDU3g3OEatJR6Ch0FDoKHaX9ThBwoyf37jmC9bBhiMVsqLxj0Uv4abNcGXNkUyVXyIjkUzmgMh9zgzJ9QlidMLBUATWoRAGVl/ICKPBbmvZXKhH13dT+1b+WlpbALdxbI+nW5tfbUu8eExQBNCAleSkvEJQbZoPYSXKFDNqJfCoHtJPMx9A3koN2MrAUtZNBJQpBafcqOorAS9QhgkJQIF8hKASFoMAVMFMPUw8EC0EhKARFdE3FlA9tAmsU1iisUSA3ISgEhaCYXcvXMotpwNRDR6Gj0FHoKDAD0ER1eTyz6dFHDn7fVPYfuRHZ6qpAgW8jHrhMGfa0jMj+Ui0oww8eBM3cYWWgwGcK9zcnIPrtcbvl7h4TFIJi4it0FAEi6hBoJ7nDyqzoKPv27Zs6dWpsbKwkSfX19RoO165dKysr69Wr17e+9a0HHnjgzJkz2lCHAUGJcFAaGxtXrFjxxhtvGECpqqrq1q3bm2++eezYsWnTpiUkJHz22WcdIqJ2EpQIB0U793pQrl271qtXr9raWnXU6/VGR0dv375dmxwcEJQ7EZT3339fkqQjR45oQGRnZz/77LNaUw3a2tpabm4ej0eSJBazAlzCu0bRzr3eUfbv3y9J0l//+ldt9OGHH545c6bWVAOns/1Xu/qNoBCUDkCho9hfqhWQYRiKQEcBU4/eYFijGLAIbkYgKGox63a7VRRaWlpYzIa6UBsMRKie8AaltbX1yPVNkiRFUY4cOXL+/Hm/319VVRUTE9PQ0HD8+PH8/Hwuj+90UPbs2aOvRiVJmjt3rt/vVy+4ffvb346Ojn7ggQf+8pe/6BNNcMzUE8pItP7wdpTgU/7legiKBkSogKC0o6WCMvRRV/r3ZVMl/NwFyezj1Ak3J0zquxDUoKUKItu2SlCJtTKo4U/IoOw/rUCUslJBlFxivRcSExQBNCAlw5+QEUrsP61AKElZqRAUNx0FYYWgEBSmHrWIuVmCaM4RKgALlEl9FyIFyqClClig2LZVCnKNYYiph8UsVM8SFIJCUOCLKlweGxJNcJOOQkeho9BRzK68BTtHqB46Ch2FjkJHoaPADEATWcyGyjhaP1NPIPUM/8/yEbPcpsrpVIDItqkG1eYa06/JqhNeODoDUcoqBVTSL9aCyhmxBtRTf3oMUVK1jChhjctyj5QSFAE0ICU5I9YglDz1p8cQSpKqZYKi8xs6Smh3ISgEhannehGDFiib0ALFtrkGKVBeODoDLFBSVimCXGMYYuphMQvVswSFoBAU6BoKQSEoBMXsgoqhEBE0mXroKHQUOgodBWYAmqje6+GVWaYeE1wIigARdYg1SjtDBIWgmHiJOqyCkjqvAvkEsf2/ahHhn5Rcd2IKqORyGVHmo25QA5YroPJ6Pg2q/uz9iOJfdCOyVZZb7u4xQRFAA1KS1/NphJL6s/cjlMS/6CYogY9ig3ay7sQUxE6Sy2XQTjIfdQvIMAwRlECNQkcxwKFvEhSCApUpBIWgEBR1SQP8k6sefZbpMKaj0FHoKICXqFPoKB26iL6TjkJHoaPQUcwu0eo9QxzTUegodBQ6Ch0FZgCayGJWnHcGLFeYegKpJ22WK2OObCrTO/LqhKHPyKBSG8pA5fVfigh/e2DFyQdBjZlWAyqlTEFkiqY6od9i672QmKAIoAEpGTOtBqEkpQwqjAYsVwhKwG9AO0ltKEPsJK//UjoKVHB8iUlqjUJHoaOYwENQBIioQ0w9LGahepagEBSCYpJwAsNMPUw9ARoEEUEhKAI8AkMEhaAEaBBEBIWgCPAIDBEUghKgQRARFIIiwCMwRFAISoAGQaSCEv/jlYnby011f7GCaPRDNaAGlSigTO9sqxPGTq0GlVgjgwr1LfbgftvL1YjAZ7PTC6z37DFBEUATDESoHoQS28vVBOWG34B2MvqhGtBOBpUodJTgtCAFdwX37Nu3b+rUqbGxsZIk1dfXaxPmzp0r6ba8vDxtqMOAqUfgJepQKP8I7reiozQ2Nq5YseKNN94IBmXSpEkXbm5///vfO+RD6yQoEQ6KdqaDQcnPz9dGTQOCcueC0q1bt549e/bv37+oqOhvf/tbMCttbW0tNzePxyNJEotZAS7BKSZUjxVTj3b6DY6yffv2hoaG48eP19fXDxw4MDMz85///Kc2WQ2cTqeujGkPCcodB4qeiffff1+SpN27d+s7/X4/HWXs1GoBGYahUP4R3B9OjmJgokePHps2bTJ06pusUQxYBDeDgQjVE66geDyeu+66q6GhQU+GISYowWQYekJhEdxvRVBaW1uPXN8kSVIU5ciRI+fPn29tbV2yZMmBAweam5t3796dnp7er1+/trY2Axz6JkExYBHcDAYiVI8VQdmzZ4+hIJ07d+7Vq1dzc3N79uzZuXNnu91eWFh48eJFPRbBMUEJJsPQEwqL4H4rghJ8yr9cjwrKoKKKIQsVU03IdiECX/WZXC6Dz6gm/WIt+Pk5xyNuUAOXKaBsW6tQ1VXZAA371XJEQ3YuttwLiQmKABqUkq0QJba6KoSSYb9aTlACLyunowigISgEhannehHDGkVcqQhcRD9ER6Gj0FHoKFz1DFmoIGvjCdkuph6mHl5HEV5TAeyEy+MAQ3QUOkqABkEaIigEhaAw9fBeT4i7P7yE334bkTcFBXd51CGCEgBlwpClucPKTGV/pRIR+LBWxhzZtrkGVPA9/Q574n/oBpXwcxeosZOrQSVVyojA45M2y2W5u8cERQANSMnYydUIJUmV5m8IV0kiKIEjBdqJbXNNh/4R3AnaSfwP3QIyDEMEhakHyj4EhaAQlHYGoE1d9bBGMaQbfZOO0k4SQdEz0WFMUAgKUw+UdggKQSEoZlfeOswyHXYy9dBR6Ch0FDoKzAA0kaueDtONvpOph6mHqQdykwAo9nJXgls2Vfr/LkeUXCGDOvthL1ATO89ClP+H+aASq2VQgxcpqJ5TBgMaOl9GNPhJ6909JigCaFBKFkGUDH5OQSgZOl8mKAG/Ae3k7Ie9EDuZ2HkWaCf5f5gvIMMwRFCYeqDsQ1AICkFhMWtW1Rryi6BJR6Gj0FHoKHQUmAFoonpllstjph4TXAiKABF1iDVKO0MEhaCYeIk6TFAICkER3fcx5UObwNQTSD19Nqyxbak2FfgJQNN/jzZhUvxzoJKqZERpT8mgXjg6A1RKmQLK9DOv6oRRM2oROb6zznKPlBIUATQgJSllCkG5YTZ0FDE0BIWgQNmHoBAUgtLSIkkSaxTWKCaLZPU6CkEhKASl4wWzgAzDkLiA1Y+yRmGNwhqFNYrZlTe9Z4hjOgodhY5CR6GjmNSx14e56jGUrsFNcbrRjzL1MPUw9VxPPYOfdCGPrw1vXIZo7O+WgEp5fTUo34VkRCNn1oKaZFsEanzT86DsG2sRDSxVEPV/rsJyd48JigAakJLxTc8jlNg31iKUDCxVCErAb0A7SXl9NWInvgvJoJ2MnFkrIMMwFMagVFRUZGRkdO3atWfPnvn5+adPn9bq1s8+++yZZ565995777777hkzZly8eFEb6jBQi1k6igEOfTOMQcnLy6urqzt58uTRo0cnT55ss9muXLmiclBUVNS3b9+mpqbDhw+PHDly1KhRHfKhdRIUPRMdxmEMinaa/X7/pUuXJEnat2+f3+/3er2dO3f+5S9/qU7485//LEnSgQMH9PMNMUHpEA59Z4SA8t5770mSdOLECb/f39TUJEnS5cuXNRpsNpuiKFpTDdra2lpubh6PR5Ikph49GYY4EkDx+XxTpkwZPXq0SsCrr74aFRWlxyIzM3Pp0qX6nn/FTqdTunUjKAY49M1IAKWoqMhut3s8ni8ECh3lTln1qFgUFxf36dPn3LlzmmGAqUebrz0pSEfRW4ghDmNHuXbtWnFxce/evc+cOaM/62ox+9prr6mdp0+fZjEb6sqKgQZBM4xBmT9/frdu3fbu3Xvh5nb16lUVjqKiIpvN9tZbbx0+fDjr+qYnKTjmqkeAiDoUxqDcWom2t+rq6lQI1Atu3bt379Kly/Tp0y9cuBAMh76HoEQyKPoz/RVjgkJQIIRUUP6j88MTo2abKq1QRvTHD2yg8E8Kj/lODSL9q8nFcfqTMqhxeVWg+jsVRD85MwLR5v8bbrm7xwRFAA1Iybi8KoSS/k4FoeQnZ0YQlIDf0FEE0BAUgsLUc72IAQuUP35go6PQUQK2IeCGoBAUghJyBSSAQz/EGiXAEB1FT4YhJigEhcUsi1ngmpvBOUI16Sh0FDoKHYWOMjFqNnKjJ61QFqyHDUMsZkPlHYtewh/2iGv4E7KpUlYpiJIqZVD3L1BA7WlORnTgAzuo+FddoHKy1oLKTVuFCHzeNuvNH1jupiBBEUADUpKTtRahJDdtFUG54TegnSRVyqCd3L9AQexkT3MyaCcHPrALyDAMEZT2H6yov0ehoxjg0DcJCkGByhSCQlAICvQzSIJCUAiK2TpZX4WIY6YeOgodhY5CR4EZgCZyeSzOO/Gvuph6mHqYeiA3ISgEhaCwRoEZgCaqNcr9j7vS5smmyrnru4jAXyOkFZrfr9ZuaKesVBBNGO8CpX1S1zTI6/UMqBGz3IjGTKtBNPLBtZa7e0xQBLiAlOT1egahZMQsN0LJmGk1BCXwpLtmGKYBYicpKxXQTiaMdwnIMAwRlEAxS0cxwKFvEhSCcuOrIXosgmOCQlAICrTiISgEhaCYrZODU0yoHqYeOgodhY5CR4EZgCbyymyojKP1M/Uw9TD1QG5CUAgKQWGNAjMATVRrlIHFFYOfV0zVf7WCKPvBalCJ1TKoQSUKoqzv1oLSShDTwP4jN6jR02sQZe9ejGhUwwLL3T0mKAJcQErsP3IjlIyeXoNQkr17MUEJ+A1oJ4nVMmIng0oU0E6yvlsrIMMwRFACxSwdxQCHvklQCAq06iEoBIWgQCsegkJQCIrZOllfhYhjph46Ch2FjkJHgRmAJvLKrDjv2LZUM/Uw9TD1QG5CUAgKQWGNAjMATfxCv3AbO7ka0aClCqikHetADV6kILK/UgnK9nI1qITt5aDAW1emb2RRJ9j+u8xyd4/BJwURSsZOrgYpGbRUASlJ2rEOoWTwIgWkxP5KJUiJ7eVqkJKE7eUE5caLDgiKGBrLgVJRUZGRkdG1a9eePXvm5+efPn1aSzPjxo2TdNvTTz+tDXUYMPWYWosYDv2o5UDJy8urq6s7efLk0aNHJ0+ebLPZrly5onIwbty4wsLCCze3lpaWDvnQOglKJIOinWa/33/p0iVJkvbt26eBsnDhQv0EcUxQ7hRQ3nvvPUmSTpw4oYHSo0eP++67LzU1tbS09B//+EcwKG1tbS03N4/HI0kSi1kBLvrkIo4tl3q0c+/z+aZMmTJ69GitZ/Pmzbt27Tp+/PjPfvazuLi46dOna0Na4HQ6dWVMe0hQIhyUoqIiu93u8Xg0CPRBU1OTJElnz57Vd/r9fjrKnbU8Li4u7tOnz7lz5wwcaM0rV65IkrRr1y6tJzhgjSLwEnVInG70o5ZLPdeuXSsuLu7du/eZM2eCz73W8/bbb0uSdOzYMa0nOCAokQzK/Pnzu3Xrtnfv3pur4AtXr171+/1nz55du3bt4cOHm5ubGxoaEhMTs7Ozg+HQ9xCUSAbFUIpKklRXV+f3+z/88MPs7Ox77703Ojo6OTm5pKSE11FCXdQ35UOboE8u4thyqUdvCV8xVh0lfWZ55qNuUw1YoSACj1ditTwh2wUq5fXViPAHwMDbEWMnVw9cpoBCf+L0Uq0dUF/3OsvdFCQoAm5ASgYuUwjKDbNB7GTACoWOYkIMYCf2l2rpKIE3GIB5Z0K2C8k7Ka+vZur5iqVIyD/OGkWQdNQhpp52eggKQQnpIvoBgkJQ9DyEjAkKQQkJh36AoBAUPQ8hY4JCUELCoR8gKARFz0PImKAQlJBw6AcICkHR8xAy9nq9kiQNnb4yfWa5qfotqUAUv8YFKntUGajkn5YiysxfBypr4hpQ/Z+vANW3uhySe11fQHHlKyRJ8nq9oU6eFGrg6+hXf1wd/LsF9ljkCIT6navf7/9GQfH5fB6Px+v13vxZfvv/qvR4PB59Z/jGYfrX8Xq9Ho/H5/OFMohvFJQOd0ItXEx/9NThn7VgZ4T9dbQjTFC0Q/HvCQjKv+c4Bv9bIuzIRthfRztft99R2tranE7nv/6p7VNYBxH219HOxe0HRdsVBlY+AgTFymfHQvtGUCx0Mqy8KwTFymfHQvtGUCx0Mqy8K7cflI0bN9rt9ujoaIfDcejQISsfLMG+Gd7ukZKSIpgcjkO3GZQdO3ZERUVt3br13XffLSwsjImJ+fjjj8PxODqdztTUVO3x7E8++SQc/xaCfb7NoDgcjuLiYnX/fD5f7969KysrBbtr2SGn0zl06FDL7t5X37HbCcrnn3/eqVOn+vp67a8xZ86cadOmac0wCpxOZ5cuXWJjYxMSEmbPnn3+/Pkw2nlkV28nKB999JEkSe+88462oyUlJQ6HQ2uGUdDY2Lhz585jx47t2rUrKyvLZrN9+umnYbT/prtKUEwP0ReecPny5XvuuWfLli1f+E9a+A/cTlAiKfUYTnFGRkZpaamhM6ybtxMUv9/vcDgWLFigHkGfzxcXFxemxawegtbW1u7du69fv17fGe7xbQZlx44d0dHR27ZtO3Xq1Lx582JiYi5evBiOx3Tx4sV79+5tbm7ev39/Tk5Ojx49Ll26FI5/kVD7fJtB8fv9GzZssNlsUVFRDofj4MGDoXbU4v0FBQWxsbFRUVFxcXEFBQXBr1G1+P6b7t7tB8V0FznBCkeAoFjhLITBPhCUMDhJVthFgmKFsxAG+0BQwuAkWWEXCYoVzkIY7ANBCYOTZIVdJChWOAthsA8EJQxOkhV2kaBY4SyEwT4QlDA4SVbYRYJihbMQBvtAUMLgJFlhF/8fU3cEBhSOMfwAAAAASUVORK5CYII="
    },
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAAGdCAIAAAAuRy6hAAAAAXNSR0IArs4c6QAAHNBJREFUeAHtnQ10FOW5x8fjMemhHgkWTs0Hs2wSCCEhQUhWvgSEaKxQ0ngLFLDk1hpIWIwoBAIIGwK7+dqZXg5oVZBwe1VSVCL2NKYtwaQVkHNjY1AR+TA33aMg9pSNoRjP6TK3YWBmMrP7zkOtMrv89/yPfed9J2Ey8+vveWdmZ5eT8MIeIOwBjrAOVsEekAAKICDtAYBC2k1YCaCAAdIeACik3YSVAAoYIO0BgELaTVgJoIAB0h74VkEJBAI+n8/v93fjZbE94Pf7fT5fIBAIRc23CorP5+PwsvAe8Pl8lgDF7/dzHBfnWZsgVpjm5y1zKUkp8RDj+OEmYhKeLqdkaM0mYpJXe4jZ9o6DmGlcHiV89WZKEjY+yXGc3+//BkHZtm2bzWaLjo52OBxHjhwJ9S9JktTd3c1xXIJYwf+yxjTFbQspGVUqEjPxwVpi+LoqSmxba4lJ2SAS8/xHE4nJuenHlAzb4qWEr97McVx3d3eow/d1S099fX1UVNTOnTs/+OCDwsLCmJiYzz77LNQ/BlBMcSFS8vxHEymU5Nz0Ywolw7Z4v3FQHA6H0+mUyQgEAnFxcZWVlQBFZxpTPpQVIhOUr7766uabb25oaFDIWLRo0ezZs5VFXQNGUYAI1YhMUD755BOO4w4dOqQAUVpa6nA4lEVJknp7e5UzQfmsB3OUUJSkbBBvXFBcLpfufBCg3HCgUEoPjGLbWssgQzcUmUaRJMnhcCxbtkyuNYFAID4+HpNZ3UwWoPThUV9fHx0dvWvXrmPHji1evDgmJubs2bPaOYq2jcmszh/GxYg1iiRJW7du5Xk+KirK4XC8/fbbWjJ0bYBiJEPXE8mg6GhgLAIUHRbGRYDSx48Myo/+8J9zDi02TfYbZZQQr8pPfLA2sUYght9RTUn2Q15iMpcKxGS8/iQxf+xMpOSeezyU3H33hm/2Ej7DH8YhgGKKC5GSjNefpFDyx85ECiX33OMBKOqNQKJOEmsEik74HdVEnWQ/5DXlQ1kBoKD0kKoPQAEoAMU4GQnRgzmKUmJCNWAUGAVGCeEPYzeMEkokSj+MAqPAKEZ1hOiBURRzhGrAKDAKjBLCH8ZuGCWUSJR+GAVGgVGM6gjRIxvlJ80LFx35mWnSl4uUTJtRSUzaCpGYnAPLKXmifQ4xI9eKxCRVC8Twz1dRQnk2qrht4c9b5lrupiBAYUBDpCSpWqBQwj9fBVCu+Iaok2kzKok6SVshUnSSc2A5USdPtM9hkKEbAijqHAVG0cGhXQQoAIU0TQEoAAWghDjHMXbjrEdbZYK2YRQYBUYxqiNED4wS1CLaThgFRoFRQvjD2A2jaOURtA2jwCgwilEdIXpglKAW0XbCKDAKjBLCH8Zu2SgTc8qn3F9lmvvGrKeE+JGQo0rFOwsFYhwLvZQYnxwO1fOD1keJ4bdXE2MXvZRkFguUpP/cbbm7xwCFAQ2REn57NYUSu+ilUJJZLAAU9VNoiTq5s1Cg6MSx0BvKH8Z+Bhm6IYCizlFgFB0c2kWAAlBI0xSAAlAAivH0JkQPznq0VSZoG0aBUWCUEP4wdsMoQS2i7YRRYBQYxaiOED0wilYeQdswCowCo4Twh7EbRglqEW0njAKjwChGdYTokY0ybs5mxwKvaYx3TIL2UJ5PltexPV1LDPGukP0lNzGP/O9PiUl2C8QQ/5YR5SIlSWUey909BigMaIiUJLsFgHJFNkH9YeyEUdjEUHQyolyEUdTPzmDvUO0oSo9x7vB1v87W+BsZPZijMIqOPITS08cPQAEoDI+oQwAFoKg0MFoABaAw8FCHAApAUWlgtAAKQGHgoQ4BFICi0sBoARSAwsBDHQIoAEWlgdGSQbHXrU36dYVpnO/MpyT9CZEY09uQygqJuzdT8sbpVGL+69gMYhxNq4lZ05FPSfJmgZLEJ633SClAYUBDpMTRtJpCyZqOfAolyZsFgKL6RhGGaYOik8Tdm4k6eeN0KoMM3RBA6atIKD06LIyLkQmKy+XiNK+UlBTGBAWgGLEw9kQsKGlpaWeuvj7//HOAErQeGYEI1ROxoGRmZrLh0I6i9ITiQ+mPWFAGDBgQGxtrt9sXLFjQ1dWlxUJu9/b2dl99+Xw+juNw1qNgYWxEJiiNjY179uzp6OhoamqaMGECz/NffPGFjhXdPAagGOHQ9kQmKFomzp8/f9ttt+3YsUPbKUkSjILTYx0SUlZWVllZmb5Xs4w5ilYeQduRb5Senp5BgwZt2bJFA4a+CVCCwqHtjExQVqxY0dLS0tnZefDgwZycnMGDB587d05Ph2YZoGiZCNqOTFDmzZsXGxsbFRUVHx8/b968U6dOaagI0gQoQeHQdkYmKEFYYHbJoGTtfWz871aZhvi49l0/8VIz33sXLSP3uijhd1QTQ38Ig3gPL3mzMK5xDSWT8msouWtWheUeKQUoLG5o7woAKKppYBS2XSg6Gde4hqKTSfk1MIqmMNHqzl3zvZS6M3Kvi1h3+B3VLIXoPr4ARlHuHqP0sLgBKACFxYfiFYACUAAK85xYM4jTY3NWYBQYxZwSN+kd8/LZEM56rpwh4/QYp8e4Mhv6E/1QelB6UHo081VmE5NZc1ZgFBjFnBJMZmXRyEZJXu0xfgqosYf4PbX8rkpi6N89nbpGpGTUKpEY239XEmP3CsTwO6soyX+riJJZv3vYcnePAQoDGiIldq9AoYTfWUWhJP+tIoCi+gZGYUADUAAKSk9p35dfEyco/K5KGAVGUbXB4AagABSAEvIMiAGHdghzFJUhGEVLhq4NUAAKJrOYzBKuuenMEWoRRoFRYBQYBUZJ2dB3jYQSxvmwbgiT2VB1B5fw1bqDC24MSiwKypTJ66dPc5smZ0IFJWkrRGKMN6hD9dz5iEAJ8SOB13TkZ//USwz9uzSL2xZSQvSofaP1PrkaoDCgASh970iR348CUAAK842QAIVQgGAUGIU0TQEoAAWgmBQcdRhzFMbsRB6CUWAUGEVVhkkLRoFRTBCRhwEKQAEorG/4M+VDWQFzlD6SYBQFiFANgAJQMJkllR0VlGEb3YnVgmmS6jdR0t6VQEz2Q15ixiwRKCHeb0uqFuyCl5ixDwvEJNYIlKQ2uCgZ8WKZ5R4pBSgMaIiUjH2YRElijUChJLXBBVBU3xB1kv2Ql6KTMUsEGIVcS65xRXkyC6PAKCbgABQGIvIQSg8ms6T5LEABKADFpOCowyg9KD0qDYwWQAEoDDzUIYACUFQaGC2AAlAYeKhDAAWgqDQwWgAFoDDwUIcACkBRaWC0ZFCm/aY458By0/C/rCGF/IHEtq21xIxwiZToHoJnLI5cKxJzb3Y5MRnLREoOdtop+f1Rm+XuHgMUBjRESu7NLqdQkrFMpFBysNMOUNQPNCDqxLa1lqKTEa5r+ORSBhm6ofAGpbW1ddasWbGxsRzHNTQ0KMXl0qVL69evv+OOO77zne/MmDHjxIkTylDQBkqPDgvjYniD0tjYuG7dur179+pAqaqqGjhw4GuvvdbR0TF79my73f7ll18GRUTuBChGMnQ94Q2Kcuy1oFy6dOmOO+6ora2VR/1+f3R09O7du5WVjQ2AosPCuBiBoJw+fZrjuPb2dgWIKVOmlJSUKItyo7e3t/vqy+fzcRyHyayRD6UnAkE5ePAgx3GffvqpQsacOXPmzp2rLMoNl8vF9X8BFAULY+PGBQVG4XdVGoEI1ROBoBBLj1YwmKOE4kPpj0BQ5Mms1+uVUeju7sZkNtTFWYUD00Z4g9LT09N++cVxnCiK7e3tXV1dkiRVVVXFxMTs27fv6NGjeXl5OD2+0UF58803+89HuYKCAkmS5Atu3//+96Ojo2fMmPHRRx9pC42xjdIT4UYxHvJ/rQegABQSOTIoGQXuOwsF0xC/hZN+B2fy7BpiiL9zxCsbieGfrSEm1KccGPv57dWUmD7jLa8wzIIfSAxQGNAYgQjVQ6GE314NUK58ORrx//22rbVEnUyeXUP8nUSdjHhlI4MM3VAoLIz9AOVKVULpMcKh7QEoAIX0kTsABaAAlO5ujuMwmdXNS7SL2uLCbsMoMAqMAqOYXVBhW0Q7CqPAKDAKjAKjUK7h4xK+dt4atK0tLuw2Sg9KD0rP5dKTvMqTsl40TW76OkqGvej+t+epD6dSQnri9fKDsaNe20AM/TNO/+PgEkps22opGVq7yXKPlAIUBjQApW8OI89RAApAMZnRAhQGIvIQjAKjkKYpAAWgABSTgqMOo/Sg9Kg0MFoABaAw8FCHAApAUWlgtAAKQGHgoQ4BFICi0sBoARSAwsBDHQIoAEWlgdECKACFgYc6JIMy8lFP2krRNMR7wvSvzFr+57nE2J6qpWR0iUjMqNUiMdOnuYlxvjOfkpRXyylJ/h/rfZ0tQGFAQ6Rk+jQ3hRLnO/MplKS8Wg5Q1O+bJupk+Z/nUnRie6qWqJPRJVSdjFotApS+AoTSw3CJPARQAArJKwAFoAAU9bTGpIXSg9Jjgog8DFAACkBhXVMx5UNZAXOUPpJgFAWIUA2AAlAwmSWVHYACUACK2U2fUIXG2I/SA6PAKNdolLTFHso3bN79h5WUjFwnEjOt+QliclNWU5IzoYKYPSfHEkN/rof4qZnJHoGSxA1uyz17DFAY0AAUtfQAFIBiUoTk6ygABaAAlOCzFgYZuiGUHpQe0nwWoAAUgGJScNRhzFF0hca4CKPAKDCKqgyTFoxiVIiuB0aBUWAUE4+owzCKzh/GRRgFRoFRVGWYtGAUo0J0PTCKapQf7180//AjphlRLlKSvlwkJqlSIGbMEoGSzKUCMfxz1cSkNriImfT7UkpMv2lZXmH4Co/l7h4DFAY0REpSG1wUSib9vhSgXPENUSfpy0WiTpIqSToZs4Sqk8ylAoMM3VB4g9La2jpr1qzY2FiO4xoaGpSZSEFBAad55ebmKkNBG/IcBUbRwaFdDG9QGhsb161bt3fvXiMo999//5mrr7/97W9B+VA6AYqWiaDt8AZFOdJGUPLy8pRR0wZACQqHtjNiQRk4cOCQIUNGjBhRVFT017/+1chKb29v99WXz+fjOA6lR0uGrh2ZoOzevXvfvn1Hjx5taGhITU3Nzs7+xz/+oWPF5XJppjF9TYCig0O7GJmgaJk4ffo0x3H79+/XdkqSBKPcQGc9yrHXzVGUfrkxePDgZ555RtepXcQcRSuPoO3IN4rP57vpppv27dunJUPXBihB4dB2hjcoPT097ZdfHMeJotje3t7V1dXT07Ny5crDhw93dnbu379/7Nixw4cP7+3t1cGhXQQoWiaCtsMblDfffFM3IS0oKLh48eJ99903ZMiQW265xWazFRYWnj17VouFsQ1QgsKh7QxvUIyH/F/rkUFJXu1J2SCaZvzcWkrsgpeYkXtdxNhfclNCeTBWXifZLRCT9OsKYogf2OxoWk3JuFeXW+6mIEBhQEOkJOnXFQDlimwoOhk/t5aoE7vgJepk5F4XRSf2l9wwyr9WWMx/CqWH4RJ5CEbpwwigABRznQAUU0qS3QKMAqOQTnwACkABKKSyA1AACkBZJrJPlSmzE5z1qBzhrMeUGMxRUHpQelRlmLRgFBjFBBF5GKAAlGsA5a6ZFZN+VGOapPpNlKStEIkh3sGxv+Tmd1WSsr2ap4X4VaEpr5bTv53Xtq2WkvTHRUpSl1rvkVKAwuAGoPQpRy49AAWgmBQggMJARB6CUWCUclNKMEe5YhoYxZQVGAVGgVFMZibqMIwCo6g0MFoABaAw8FCHAApAUWlgtAAKQGHgoQ4BFICi0sBoARSAwsBDHQIoAEWlgdGSQRnq3WR7qtY0pvtUXsH2dC0xgTPJxEzLqaTE+c58YrSPobPbox8TiSHeMx9VKlKSUmK9u8cAhcEKkZLRj1HfXEGhZFSpCFBU3xB1EjiTTNHJtJxKok6c78xnkKEbAih9FQmlR4eFcRGgABTS9yYAFIACUBjnOf2HUHqMtUbXA6PAKDBKf20wlmAUnT+MizAKjAKjMBzSfwhGMSpE1wOjwCgwSn9tMJZgFJ0/jIswimqUYRXuxBrBNMSbFKRnPy8/Ijr13kpihr3gpmR0iUgM/WJ/6hqRGCNkQXvG/UygZMxCt+U+kBigMLghUpK6RgyKhbGTQsm4nwkARX3cnKiTqfdWUnQy7AU3USejS0QGGbohgILSQ3rzCkABKACFcZ7Tf0g+68EcRVdutIswCowCo/TXBmMJRtHKI2gbRoFRYBSGQ/oPwShBLaLthFFgFBilvzYYSzCKVh5B2zAKjAKjMBzSfwhGCWoRbSeMoholo8B9Z6FgGuI3AM85tJiYvD8VE9P7qZ2SDKdIzOTZNcQk7t5MzPCXN1KSVC1QYt9ovbvHAIUBDZGSxN2bKZQMf3kjhZKkagGgqL4h6iTvT8UUnfR+aifqJMMpMsjQDYUxKB6PJysr69Zbbx0yZEheXt7x48eViceXX365dOnS22+//bvf/e6DDz549uxZZShoQ56jwCg6OLSLYQxKbm5uXV3d+++//+677z7wwAM8z1+4cEHmoKioaOjQoc3NzW1tbePHj584cWJQPpROgKJlImg7jEFRDrMkSefOneM4rrW1VZIkv99/yy23vPzyy/IKH374Icdxhw8f1q6vawOUoHBoOyMElJMnT3Ic995770mS1NzczHHc+fPnFRp4nhdFUVmUG729vd1XXz6fj+M4lB4tGbp2JIASCARmzpw5adIkmYAXX3wxKipKi0V2dvaqVau0Pf9su1wurv8LoOjg0C5GAihFRUU2m83n810TKDDKjXLWI2PhdDoTEhI+/vhjRRjE0qOsr3yQDoyiVYiuHcZGuXTpktPpjIuLO3HihPaoy5PZV155Re48fvw4JrOhrqzoaGAshjEoxcXFAwcObGlpOXP1dfHiRRmOoqIinucPHDjQ1tY24fJLS5KxjbMeBiLyUBiD0n8m2rdUV1cnQyBfcBs0aNCAAQPy8/PPnDljhEPbA1AiGRTtkf6abYACUEgIyaCMv79i8g9rTJNYLVDS3pVAzLBfeInJfshLSfKeCmIyiwVikqoEYuyCl5Lnjk+mZEvbeMs9ewxQGNAQKUmqEiiU2AUvhZLnjk8GKKpviDoZ9guSTrIf8hJ1kryngkGGbgig9NUmlB4dFsZFgAJQSNMUgAJQAArpjAegABSAYnaebJyLhOpB6YFRYBQYBUYhM0BaEafHoSqO0o/Sg9KD0kOyCUABKNcISupST/rjommIj7jx26uJoVv9jdOplBz+PxsxY3+7lpi7Z1UTM/ZhgZLclhJKZvx2ieVuCgIUBjRESu6eVU2hZOzDAoWS3JYSgKL6BkZhQANQAApKz+UPdCBOUPjt1TAKjKJqg8ENQAEoACXkGRADDu0Q5igqQzCKlgxdG6AAFExmMZklXHPTmSPUIowCo8AoMAqMkv64iHs97Mv5uIR/5X4hQLnRQRm+wjNyrWia3LS1lKQ8KRKTtlIkJqlSoMSx0EsM/TMK7pnuIWbUKpGSMUUCJaMftt4HEgMUBjdESu6Z7qFQMmqVSKFkTJEAUFTfEHWStlKk6CSpUiDqxLHQyyBDNwRQ1He4wSg6OLSLAAWgkL4NAaAAFIByje+ZRenR1hpdG0aBUWAUGMXsgopOG4xFGAVGgVFgFBiFzABpRfnZY0xmUXpMcAEoDETkIcxR+hgCKADFxCXysAxK4jr38E2CaexegRLTr8VVVuCfrSEm2SNQkrFMJIb+QaP8jmpiRpeIlEw/8DglU36z1HLPHgMUBjdESvgd1RRKRpeIFEqmH3gcoKjfxE3UCf9sDUUnyR6BqJOMZSKDDN0QQFHnKDCKDg7tIkABKKRvWAAoAAWgyKc0hP/irEdbZYK2YRQYBUYhuEReBUYJahFtJ4wCo8AoMIrZJVqtM9htGAVGgVFgFBiFzABpRUxm2XUneU8FSo9aelKLPenLRdMod33ZDf6XNcSkNriIySoQKLH9ykMM8d9NbXCNa1xDjP0lNyXTmp+gZPLrTsvdPQYoDG6IlIxrXEOhxP6Sm0LJtOYnAIrqG8bh0Q1RdJJVIBB1YvuVR/f7GYthDIrH48nKyrr11luHDBmSl5d3/PhxZT4ydepUTvNasmSJMhS0Ic9RYJTIBCU3N7euru79999/9913H3jgAZ7nL1y4IHMwderUwsLCM1df3d3dQflQOgEKAxF5KIyNohxmSZLOnTvHcVxra6sCymOPPaZdgd0GKDcKKCdPnuQ47r333lNAGTx48Pe+9720tLSysrK///3vRlB6e3u7r758Ph/HcSg9DFwiwSiBQGDmzJmTJk1SaHj22WebmpqOHj36wgsvxMfH5+fnK0NKw+VyaaYxfU2AEuGgFBUV2Ww2n8+nQKBtNDc3cxx36tQpbackSTDKjXLWIx94p9OZkJDw8ccf6zhQFi9cuMBxXFNTk9JjbGCOwnCJPBTGpefSpUtOpzMuLu7EiRPGY6/0vPXWWxzHdXR0KD3GBkCJZFCKi4sHDhzY0tJy9Sz4zMWLFyVJOnXqVEVFRVtbW2dn5759+xITE6dMmWKEQ9sDUCIZFN1UlOO4uro6SZL+8pe/TJky5fbbb4+Ojk5OTi4tLcV1lFDXak35UFYI49KjVcLXbMtGSX/EnblUMI3pM6fyCrana4nJeFQkhnh/xPS+prJChlMkRvkR0wb/fBUpdVU8IQlPl1vupiBAYUBjyoeyAomS50mU8HVVAEX1DVEnGY+KMIqxdHDGrm+uB6WH4RJ5SBGGaQNGuTJ9wRyFzQpAASjmb/9LXy4CFIACULq7OY7DWQ9jpsIuN9pRGAVGgVFgFLMrb1pnsNswCowCo8AoMArlMh0uuDGmsbjgpiLk9/v73gq5aH36I27TJK5zUzJU2ERM2hIPMZNfd1KSWuwhJq3QQwzxF6YWexK2lZPydHkCIXFCGcdxfr9fPVr9W9/qJXz5zdXG9y2gxyJ7INT7XCVJ+lZBCQQCPp/P7/dffVt+3//K9Ph8Pm1n+LbD9M/x+/0+ny8QCPT3iLr0rYKi/rOaljxxMX3Tk+YnLN2MsD9H2dcARdkV/54GQPn37Efjb4mwPRthf45yvK6/UXp7e10u1z//q2xTWDci7M9RjsX1B0XZFDSsvAcAipWPjoW2DaBY6GBYeVMAipWPjoW2DaBY6GBYeVOuPyjbtm2z2WzR0dEOh+PIkSNW3lmMbdN9ukdKSgpj5XAcus6g1NfXR0VF7dy584MPPigsLIyJifnss8/CcT+6XK60tDTl8ezPP/88HP8KxjZfZ1AcDofT6ZS3LxAIxMXFVVZWMjbXskMulyszM9Oym/f1N+x6gvLVV1/dfPPNDQ0Nyp+xaNGi2bNnK4th1HC5XAMGDIiNjbXb7QsWLOjq6gqjjads6vUE5ZNPPuE47tChQ8qGlpaWOhwOZTGMGo2NjXv27Ono6GhqapowYQLP81988UUYbb/ppgIU0110zSucP3/+tttu27FjxzX/pIV/4HqCEkmlR3eIs7KyysrKdJ1hvXg9QZEkyeFwLFu2TN6DgUAgPj4+TCezWgh6enoGDRq0ZcsWbWe4t68zKPX19dHR0bt27Tp27NjixYtjYmLOnj0bjvt0xYoVLS0tnZ2dBw8ezMnJGTx48Llz58LxDwm1zdcZFEmStm7dyvN8VFSUw+F4++23Q22oxfvnzZsXGxsbFRUVHx8/b94848eoWnz7TTfv+oNiuolYwQp7AKBY4SiEwTYAlDA4SFbYRIBihaMQBtsAUMLgIFlhEwGKFY5CGGwDQAmDg2SFTQQoVjgKYbANACUMDpIVNhGgWOEohME2AJQwOEhW2ESAYoWjEAbbAFDC4CBZYRP/H0x+yTldUeiDAAAAAElFTkSuQmCC"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![image.png](attachment:image.png) ![image-2.png](attachment:image-2.png)  \n",
    "我们可以看到最后整个C词嵌入表是变了的,因为我们在输入的时候将C view给神经网络并且反向传播也包括C,虽然每次只传入了三个字符,但是变化是记录在C内的,下面演示一下\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.tensor(range(10))\n",
    "print(a)\n",
    "b = a.view(2,5)\n",
    "b[1,:]=0\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "深度神经网络的参数初始化的问题\n",
    "就像前面看到的,许多激活函数不希望输出的方差太大,那么如何设置权重来使得在计算过程中输出不会太分散呢?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.randn((1000,10))\n",
    "# !!!!!\n",
    "w = torch.randn((10,200)) / 10**0.5\n",
    "# !!!!!\n",
    "y = x @ w\n",
    "print(x.mean(),x.std())\n",
    "print(y.mean(),y.std())\n",
    "plt.figure(figsize=(20,5))\n",
    "plt.subplot(121)\n",
    "plt.hist(x.view(-1).tolist(),50)\n",
    "plt.subplot(122)\n",
    "plt.hist(y.view(-1).tolist(),50)\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结一下\n",
    "\n",
    "- 更好的参数初始化保持高斯分布的范围\n",
    "- 直接对隐藏层进行高斯分布来保持"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa = torch.randint(0, 100, (1000,)).float()\n",
    "aamean = aa.mean()\n",
    "aastd = aa.std()\n",
    "plt.figure()\n",
    "bb = ((aa-aamean)/aastd)\n",
    "plt.subplot(121)\n",
    "plt.hist(aa.tolist(), 50)\n",
    "plt.subplot(122)\n",
    "plt.hist(bb.tolist(), 50)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3.7.2",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
